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ABSTRACT 


The  methods  for  evaluating  Air  Force  engineering  proposals 
are  well  established.  However,  at  times,  the  rating  a  proposal 
receives  is  arbitrarily  and/or  subjectively  determined.  The 
method  proposed  in  this  study  will  yield  a  systematically 
determined  rating  for  each  proposal . 

In  this  thesis,  a  multi-attribute  evaluation  methodology  is 
developed  based  on  the  Analytical  Hierarchy  Process  to  prioritize 
proposals.  The  methods  are  applied  to  two  different  stages  of 
the  Air  Force  laboratory  procurement  process:  the  first  stage  is 
the  resource  allocation  phase  where  division- 1 evel  management 
decides  which  projects  will  be  funded;  the  second  stage  is 
proposal  evaluation,  where  the  project  engineer  evaluates 
proposals  sent  in  by  defense  contractors. 

The  methodology  used  in  both  stages  follows  a  two  step 
process:  first,  data  on  preferences  among  the  various  attributes 

is  gathered  through  ^■he  use  of  pairwise  comparisons  and  a 
questionnaire;  second,  the  pairwise  comparison  data  are  used  to 
determine  the  relative  weights  of  all  the  attributes. 

The  hierarchies  in  both  stages  of  this  study  were  developed 
using  attributes  that  have  been  used  in  previous  resource 
allocation  and  proposal  evaluation  cases.  The  hierarchies  were 
then  used  to  determine  the  "best”  projects  to  fund  in  the 
resource  allocation  stage,  or  proposals  in  the  proposal 
evaluation  stage. 

The  results  of  this  study  led  us  to  the  conclusion  that  a 
mul ti -attribute  evaluation  methodology  based  on  the  analytical 
hierarchy  process  can  be  readily  applied  to  both 
stages  of  the  procurement  process  studied  and  provide  an 
objective,  systematic  means  for  prioritizing  projects  and 
proposals . 
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ABSTRACT 


The  methods  for  evaluating  Air  Force  engineering 
proposals  are  well  established.  However,  at  times,  the 
rating  a  proposal  receives  is  arbitrarily  and/or 
subjectively  determined.  The  method  proposed  in  this  study 
will  yield  a  systematically  determined  rating  for  each 
proposal . 

In  this  thesis,  a  multi -attribute  evaluation 
methodology  is  developed  based  on  the  Analytical  Hierarchy 
Process  to  prioritize  proposals.  The  methods  are  applied  to 
two  different  stages  of  the  Air  Force  laboratory  procurement 
process:  the  first  stage  is  the  resource  allocation  phase 

where  division- level  management  decides  which  projects  will 
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be  funded;  the  second  stage  is  proposal  evaluation,  here 
the  project  engineer  evaluates  proposals  s^nt  in  by  defense 
contractors . 

In  the  resource  allocation  stage,  a  total  of  twenty- 
five  attributes  was  used,  and  in  the  proposal  evaluation 
stage  twenty- three  attributes  were  used.  The  methodology 
used  in  both  stages  follows  a  two  step  process:  first,  data 
on  preferences  among  the  various  attributes  is  gathered 
through  the  use  of  pairwise  comparisons  and  a  questionnaire; 
second,  the  pairwise  comparison  data  are  used  to  determine 
the  relative  weights  of  all  the  attributes.  The  weights  are 
used  to  rank  the  projects/proposals. 

The  hierarchies  in  both  stages  of  this  study  were 
developed  using  attributes  that  have  been  used  in  previous 
resource  allocation  and  proposal  evaluation  cases.  The 
hierarchies  were  then  used  to  determine  the  "best"  projects 
to  fund  in  the  resource  allocation  stage,  or  proposals  in 
the  proposal  evaluation  stage,  using  actual  laboratory 
procurement  cases. 

The  results  of  this  study  led  us  to  the  conclusion  that 
a  multi -attribute  evaluation  methodology  based  on  the 
analytical  hierarcy  process  can  be  readily  applied  to  both 
stages  of  the  procurement  process  studied  and  provide  an 
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objective,  systematic  means  for  prioritizing  projects  and 
proposals . 

Umesh  K.  Saxena,  Major  Professor  Date 
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I .  INTRODUCTION 


"At  the  very  least,  10  percent  of  the  $76  billion  that  the 
government  now  spends  each  year  on  research  should  be 
redirected  from  the  Pentagon*  s  research  budget  to  civil 
efforts."  President  Bill  Clinton. 

1.1.  Baekgyound 

Even  before  Mr  Clinton  committed  these  words  to  paper 
in  his  1992  Presidential  Campaign  literature  entitled 
Technology:  The  Engine  of  Economic  Growth,  the  defense 

budget,  and  hence  the  money  available  for  defense  related 
research,  has  been  steadily  falling  since  the  peak  of  the 
Reagan  build-up.  In  fact,  with  the  Fiscal  1993  budget,  for 
the  eighth  consecutive  year,  the  Pentagon's  spending  power 
will  decline  (Green,  1992).  Now,  with  Bill  Clinton's 
election,  exactly  how  far  the  defense  budget  will  tumble  is 
unknown . 

With  the  demise  of  the  Soviet  Union  and  the  end  of  the 
cold  war,  the  perceived  need  for  the  military  has 
diminished.  Congress  appears  determined  to  continuously 
slash  defense  and  spend  the  "peace  dividend"  on  other 
programs  in  their  respective  districts. 

Future  defense  budgets  will  continue  to  decline.  By 
1997,  it  is  estimated  that  US  defense  spending  as  a 
percentage  of  Gross  National  Product  and  of  the  federal 
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budget,  will  be  lower  than  its  post-  World  War  II  levels 
(Canan,  1992  and  Auster,  1993).  The  chart  in  Figure  1.1.1 
shows  where  the  defense  budget  will  absorb  the  cuts;  notice 
that  Research  and  development  will  drop  by  23.8  percent. 
(Auster,  1993) 


FIGURE  1.1.1. 


Outlay  Reductions  1988-97 

Budget  Category 

Dollar  Drc^ 

Percent  Drop 

Military  procurement 

$46  billion 

46.0 

Military  personnel 

25  billion 

28.4 

Operations  and  me.intenance 

19  billion 

19.8 

Research  and  development 

10  billion 

23.8 

Military  construction 

1  billion 

9.1 

Total 

$101  billion 

One  important  lesson  of  the  Persian  Gulf  War  was  that 
high  technology  weapon  systems  can  become  significant  force 
multipliers,  and  more  importantly,  can  save  the  lives  of 
American  forces.  Therefore,  it  is  imperative  that  US  forces 
continue  to  develop  the  most  technologically  advanced 
systems  in  the  world,  in  spite  of  these  budget  cuts. 

To  maintain  technological  superiority  in  the  days  of 
austere  budgets,  procurement  and  research  and  development 
(R  &  D)  organizations  will  have  to  become  even  more 
efficient  than  they  already  are.  One  method  being  discussed 
in  Washington  to  accomplish  this  is  to  emphasize  prototyping 
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weapons.  In  an  Air  Force  magazine  article,  former  Deputy 
Secretary  of  Defense  Donald  Atwood  had  this  to  say  about  the 
situation,  "Production  rates  will  be  considerably  less  than 
in  the  past  and  below  the  capacity  of  industry  to  produce. 

It  is  vitally  important  to  push  the  research  and  development 
of  innovative  new  technology."  Thus,  research  and 
prototyping  take  on  added  significance.  {Auster,  1992) 

1.2.  Purpose  of  Study 

The  purpose  of  this  study  is  to  illustrate  how  a 
multiattribute  evaluation  model  based  on  the  Analytical 
Hierarchy  Process  can  be  used  for  resource  allocation  and 
proposal  evaluation  in  Air  Force  R  &  D  organizations.  Where 
appropriate,  key  attributes  applicable  to  exploratory  and 
advanced  development  research  in  both  the  resource 
allocation  and  proposal  evaluation  processes  will  be 
identified.  The  attributes  and  hierarchies  developed  in 
this  study  are  unique  to  the  organization  examined.  The 
specific  objectives  of  the  study  are  to: 

a.  Establish  an  objective  resource  allocation 
procedure; 

b.  Develop  an  universally  acceptable  methodology  for 
proposal  evaluation; 

c.  Assign  a  quantified  numerical  output  to  identify 
resource  allocation  priority  and  individual 
proposal  merit;  and 


d.  Compare  and  contrast  the  present  methods  with  the 


proposed  ones . 

1.3.  Slonlf Icance  of  the  Study 

Project  engineers  and  program  managers  in  Air  Force  R  & 
D  organizations  are  directly  involved  in  the  allocation  of 
large  sums  of  money  entrusted  to  them  by  the  American 
people.  This  money  will  be  used  to  accomplish  objectives 
set  forth  by  the  highest  elected  officials  in  the  land, 
those  designated  as  being  vital  to  the  defense  of  the  United 
States.  Many  of  the  programs  will  have  important  long-term 
implications  to  the  nation.  (Laboratory  Program  Managers 
Guide,  1985) 

The  proper  evaluation  of  research  proposals  and 
subsequent  selection  of  a  contractor  to  conduct  the  research 
plays  an  important  role  in  the  management  of  military 
technology.  With  the  defense  budget  continuously  shrinking, 
and  with  the  present  push  towards  prototyping  systems,  the 
appropriate  selection  of  contractors  to  perform  R  &  D  takes 
on  added  significance. 

1.4  Organization  of  the  Study 

This  study  is  divided  into  five  sections.  The  first 
section  introduces  the  thesis  and  provides  background 
highlighting  the  need  for  formal,  scientific  procedures  to 
allocate  resources  and  perform  proposal  evaluations  in  Air 
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Force  R  &  D  organizations.  The  second  section  contains  an 
examination  of  the  present  processes  and  a  literature 
review,  it  outlines  descriptions  of  the  present  process  for 
allocating  resources  in  the  Electronic  Warfare  Division  of 
the  Avionics  Directorate  at  Wright  Patterson  Air  Force  Base, 
Ohio  and  generally  accepted  procedures  for  evaluating  R  &  D 
proposals.  Also  in  the  second  section  is  a  review  of 
related  research  work.  A  detailed  discussion  of  the 
proposed  method  is  presented  in  the  Section  Three.  Included 
in  this  section  is  the  development  of  the  hierarchies  for 
projects  and  proposals,  the  construction  of  questionnaires 
to  gather  information,  the  development  of  priority  weights, 
and  the  formulation  of  the  resource  allocation  and  proposal 
evaluation  methods.  The  methodology  was  applied  to  existing 
resource  allocation  and  proposal  evaluation  data,  and  the 
results  of  these  tests  are  presented  in  the  Section  Four. 
Detailed  discussion  of  the  outcomes  and  calculations  are 
also  included  in  this  section.  The  fifth  section  contains 
additional  discussions  and  conclusions  along  with 
recommendations  for  future  work. 
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II.  EXAMINATION  OF  EXISTING  PROCESS  AND  LITERATURE  REVIEW 


There  are  already  methods  in  existence  for  resource 
allocation  and  proposal  evaluation  in  the  organization 
studied.  To  better  understand  how  the  proposed  methods  can 
be  applied,  it  will  be  beneficial  to  briefly  discuss  the 
methods  presently  used;  therefore,  an  explanation  of  the 
organization  and  the  present  processes  is  presented  here. 

An  extensive  review  of  literature  revealed  that 
different  organizations  in  the  Department  of  Defense  (DOD) 
are  examining  ways  to  utilize  multiattribute  decision  models 
in  the  project  and  source  selection  processes.  Results  of 
some  of  these  studies  are  presented  in  this  section. 

2.1.  Existing  Laboratory  Procurement  Process 

The  following  paragraphs  will  describe  the  procurement 
process  used  in  the  Avionics  Directorate's  Electronic 
Warfare  Division  at  the  time  of  the  study.  A  brief 
explanation  of  the  organizational  structure  and  how  the 
division  receives  its  budget  is  presented. 

2.1.1.  Organizational  Structure 

The  Electronic  Warfare  Division  is  part  of  the  Avionics 
Directorate  at  Wright  Patterson  Air  Force  Base  near  Dayton, 
Ohio.  Figure  2.1.1.  shows  the  division's  position  in  the 
directorate's  hierarchy. 
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As  can  be  seen  in  Figure  2.1.2.,  the  division  is 
comprised  of  four  branches;  the  Requirements  and 
Effectiveness  Evaluation  Branch,  the  Electronic 
Countermeasures  Advanced  Development  Branch,  the  Passive 
Electronic  Countermeasures  Branch,  and  the  Active  Electronic 
Countermeasures  Branch.  Each  of  these  branches  will  have 
several  groups  underneath  them. 

The  division  conducts  basic,  exploratory,  and  advanced 
development  research  in  electronic  warfare  techniques  and 
technologies .  Research  and  development  is  conducted  both 
in-house  and  through  defense  contractors.  Areas  of  research 
include  but  are  not  limited  to:  radio  frequency  (RF) , 
electro-optical (EO) ,  and  infrared  (IR)  electronic  warfare 
techniques  and  systems;  electronic  warfare  sensors, 
receivers,  and  transmitters;  and  foreign  threat  systems  and 
capabilities.  Figure  2.1.2.  also  provides  a  brief  sample  of 
some  of  the  technologies  being  worked  by  each  branch. 

2.1.2.  Resource  Allocation  Process 

The  budget  the  division  uses  to  fund  new  programs  is  a 
part  of  the  overall  annual  defense  budget.  Describing  the 
in-depth  process  of  how  money  is  allocated  to  the  division 
through  the  DOD  Biennial  Planning  and  Budgeting  System  is 
beyond  the  scope  of  this  study;  however  a  brief  description 
will  prove  beneficial.  The  budgetary  process  is  highlighted 
in  Figure  2.1.3. 
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In  general,  the  Electronic  Warfare  Division  receives 
money  from  three  sources:  1)  the  program  element  (PE)  in 
the  overall  DOD  budget  for  Avionics,  2)  the  PE  for 
Electronic  Warfare  Technology,  and  3)  outside  users.  The 
dollars  from  the  Avionics  PE  funds  exploratory  development 
research,  while  the  money  from  the  Electronic  Warfare  PE  is 
applied  to  advanced  development  projects.  The  funds  from 
outside  users  are  applied  to  special  projects  on  an  as 
needed  basis. 

Each  year,  after  the  President  signs  the  budget,  the 
division  will  know  how  much  money  they  have  from  the  PEs 
described  above.  The  money  owed  on  previously  funded 
programs  in  the  given  fiscal  year  is  subtracted  from  the 
money  allocated  through  the  PEs,  this  gives  the  amount  of 
money  that  is  available  to  fund  new  programs. 

To  determine  which  new  programs  will  be  funded,  the 
division  convenes  the  Resource  Allocation  Committee  (RAC) 
each  January.  The  RAC  is  made  up  of  the  Division  Chief,  the 
Technical  Director  for  the  division,  and  the  four  Branch 
Chiefs.  Project  engineers  from  the  groups  will  brief  their 
ideas  for  new  programs  to  the  RAC. 

For  each  program  briefed  to  the  RAC,  each  member  is 
given  a  rating  sheet  that  contains  a  brief  program 
description,  funding  information  on  the  program,  and  space 
for  the  evaluators  ratings  of  the  program  against  various 


criteria . 


11 


FIGURE  2.1.4. 
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An  evaluator  will  use  the  criteria  and  weights  in 
Figure  2.1.4  to  determine  his  "total"  score  for  an  analysis 
program.  After  all  programs  are  presented  and  similarly 
scored,  the  evaluator  will  rank  the  projects  in  ascending 
order  from  one  through  the  number  of  projects  briefed,  where 
one  means  this  is  the  best  program  in  the  evaluators  eyes, 
two  is  the  second  best,  etc.  The  individual  rankings  given 
by  each  RAC  member  for  a  given  program  are  then  averaged  to 
give  the  program  its  final  score.  All  the  programs  are  then 
arranged  in  ascending  order  according  to  their  final  score, 
money  is  allocated  to  the  programs  with  the  highest  final 
scores  until  all  available  money  for  the  given  fiscal  year 
has  been  allotted.  The  rankings  given  by  each  member  are 
averaged  to  remove  any  bias  (positive  or  negative)  the  RAC 
member  may  have  toward  a  program.  The  results  of  a  recent 
analysis  RAC  are  presented  in  Figure  2.1.5. 

2.1.3.  Request  for  Proposal  Process 

Once  a  project  engineer  is  notified  that  his  new 
program  idea  will  be  funded,  he  begins  preparing  a  purchase 
request  package  (PRP) .  The  PRP  forms  the  basis  of  the 
solicitation  and  award  phase  of  the  procurement  process. 

This  package  authorizes  the  procurement  office  to  solicit 
proposals  for  the  award  of  R  &  D  contracts,  and  describes 
the  technical  requirement  and  other  information  to 
administer  and  control  the  remainder  of  the  procurement 
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process.  This  package  contains  many  things  including  the 
Statement  of  Work,  which  thoroughly  describes  the  research 
effort  and  prescribes  what  is  expected  of  prospective 
contractors,  and  the  evaluation  factors  and  criteria  that 
will  be  applied  to  the  contractor's  proposals.  The 
evaluation  factors  and  criteria  will  communicate  to 
potential  offerors  the  important  considerations  which  will 
be  used  in  the  evaluation  of  proposals.  (AFR  70-30,  par  17b) 

Through  various  methods,  a  list  of  possible  sources  for 
the  research  is  obtained.  Those  contractors  on  the  list 
will  receive  a  formal  Request  for  Proposal,  which  will 
contain  the  Statement  of  Work  and  evaluation  criteria. 

Those  contractors  who  elect  to  will  send  their  proposals  to 
the  Air  Force  contracting  office  for  the  requesting  R  &  D 
organization.  The  proposals  will  be  forwarded  to  the 
responsible  project  engineer  for  technical  evaluation. 

2.1.4.  Technical  Evaluation  Process 

Technical  evaluation  of  proposals  in  Air  Force  R  &  D 
organizations  is  accomplished  in  accordance  with  policies 
and  procedures  set  forth  in  Air  Force  Regulation  (AFR)  70-30 
and  WRDC  Regulation  70-5.  These  evaluations  are  performed 
by  a  team  of  evaluators,  with  the  project  engineer  serving 
as  Team  Chief . 

The  proposals  are  rated  against  the  criteria 
established  by  the  evaluation  team  and  sent  to  the 
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prospective  contractors  in  the  Request  for  Proposal  package. 
APR  70-30  describes  the  requirements  for  the  evaluation 
criteria: 

Specific  criteria  relate  to  program  or  project 
characteristics.  The  specific  criteria  are  divided 
into  appropriate  technical  and  (or)  management 
evaluation  areas.  Areas  of  specific  criteria  are 
evaluated  in  a  matrix  fashion  against  assessment 
criteria.  Examples  of  specific  criteria  might  include 
technical,  logistics,  manufacturing,  operational 
utility,  design  approach,  readiness  and  support,  test 
and  management.  These  areas  are  further  subdivided 
into  items,  factors,  and  in  some  instances,  subfactors. 
The  lowest  level  of  subdivision  depends  on  the 
complexity  of  the  area  being  evaluated,  (APR  70-30) 

These  official  guidelines  for  the  evaluation  criteria 
are  consistent  with  the  establishment  of  a  hierarchy  for 
proposal  evaluation  in  this  study  using  AHP. 

The  technical  team  will  rate  each  proposal  against  the 
lowest  level  criteria  as  either  Exceptional,  Acceptable, 
Marginal,  or  Unacceptable.  Except  for  the  rating 
Unacceptable,  each  rating  can  have  a  (+)  or  {-)  associated 
with  it  indicating  there  is  a  high,  middle,  and  low  range 
for  each  rating.  The  definitions  for  these  ratings  are 
contained  in  APR  70-30,  and  are  presented  in  Table  2.1.1. 


The  technical  team  is  required  to  prepare  a  written 
report  that  documents  the  results  of  the  evaluation  of  the 
proposals  against  the  standards.  The  report  is  sent  to  the 
contracting  office  and  becomes  the  basis  for  negotiations 
between  the  Air  Force  and  the  contractors . 


TABLE  2.1.1. 


Rating 

Definition 

Exceptional 

Exceeds  specified  performcince  or  capcJaility 
in  a  beneficial  way  to  the  Air  Force;  and 
has  high  probability  of  satisfying  the 
requirement;  and  has  no  significant  weakness 

AcceptcJole 

Meets  evaluation  standards ;  and  has  good 
probability  of  satisfying  the  requirement; 
and  any  weaknesses  can  be  readily  corrected 

Marginal 

Fails  to  meet  minimiom  evaluation  standards; 
cuid  has  low  probability  of  satisfying  the 
requirement;  and  has  significant 
deficiencies  but  correctable 

Unacceptcible 

Fails  to  meet  a  minimum  requirement;  and 
deficiency  requires  major  revision  to  the 
proposal  to  make  it  correct 

The  evaluation  report  will  contain  an  overall  rating 
for  each  contractor's  proposal.  Additionally,  the  report 
will  include  a  discussion  of  the  acceptable  proposals 
considering  each  of  the  evaluation  criteria  with  a  narrative 
describing  how  the  offeror  met  the  criteria.  If  any 
proposal  is  found  to  be  unacceptable,  this  must  be  fully 
documented  in  the  report . 
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2.2.  Review  of  Related  Work 

Much  has  been  written  about  the  advantages  and 
disadvantages  of  the  application  of  the  Analytical  Hierarchy 
Process  (AHP) .  It  is  not  within  the  scope  of  this  study  to 
discuss  the  arguments  (either  for  or  against)  related  to 
AHP's  application.  Therefore,  the  literature  review  focused 
on  applications  of  AHP  in  the  research  and  development  (R  & 
D)  and  source  selection  environments. 

Section  2.1  above  described  how  the  existing  processes 
for  resource  allocation  and  proposal  evaluation  work;  the 
literature  review  concentrated  on  alternatives  to  these 
procedures.  It  was  discovered  that  various  multiattribute 
decision  techniques  have  been  examined  by  others  in  DOD  to 
see  how  they  may  be  utilized  in  various  stages  of  the 
procurement  process.  The  results  of  some  of  the 
representative  analyses  will  be  described  in  this  section. 

2.2.1.  Decision  Analysis  in  Source  Selection 

Two  professors  at  the  United  States  Air  Force  Academy, 
Lieutenant  Colonel  Vernon  E.  Francis  and  Captain  Jeffrey  S. 
Stonebraker  investigated  the  application  of  formal  decision 
analysis  methods  to  the  selection  of  the  "best"  contractor 
for  defense  system  acquisition  (Francis  and  Stonebraker, 
1991) .  They  examined  Multiattribute  Utility  Theory  (MAUT) 
and  the  AHP  to  determine  how  they  could  be  applied  to  source 
selection  in  an  Air  Force  system  program  office. 
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Additionally,  they  described  the  source  selection  process 
and  both  MAUT  and  AHP,  as  well  as  apply  them  both  to  an 
actual  air  Force  source  selection  case  study.  Because  this 
study  focuses  on  the  AHP,  only  their  findings  related  to  it 
will  be  discussed. 

With  current  source  selection  procedures  in  systems 
acquisition,  evaluation  criteria  are  broken  down  into  areas 
and  areas  are  further  subdivided  into  items,  factors,  and 
subfactors  (AFR  70-30)  .  The  authors  found  that  these 
elements  map  directly  into  a  hierarchical  model  for  use  in 
AHP.  In  their  study,  they  used  the  application  of  AHP  found 
in  the  software  package  EXPERT  CHOICE  to  aid  in  analysis  of 
the  source  selection  problem. 

Because  current  procurement  directives  prohibit  the 
direct  comparison  of  competing  contractors'  proposals,  an 
absolute  system  of  measurement  had  to  be  developed.  These 
gentlemen  created  an  interesting  absolute  measurement  system 
that  will  be  described  in  detail  and  applied  later  in  this 
study. 

They  also  investigated  ways  to  make  the  application  of 
AHP  to  source  selection  more  consistent  with  established 
procedures  so  that  evaluators  and  decision  makers  would  not 
have  to  learn  a  drastically  different  method.  They  did  this 
by  translating  the  numerical  ratings  produced  by  AHP  into 
the  color  ratings  used  in  the  present  evaluation  system. 
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In  their  study  of  the  application  of  AHP  to  source 
selection,  they  arrived  at  the  following  conclusion: 

It  is  the  conclusion  of  this  research  that  formal 
decision  analysis  should  be  used  to  improve  the  source 
selection  process.  In  particular,  decision  analysis 
would  assist  evaluators  in  structuring  this  important 
problem,  ascertaining  the  criteria  upon  which  the 
decision  should  be  based,  deriving  weights  for  these 
criteria,  and  evaluating  contractor  proposals  against 
these  criteria.  Moreover,  decision  analysis  would 
facilitate  a  more  logical  and  consistent  synthesis  of 
judgements  into  a  final  ranking  of  contractor 
proposals.  (Francis  and  Stonebraker,  1991) 

2.2.2.  Decision  Models  for  Army  R  &  D  Project  Selection 

Given  the  recent  changes  in  the  international  strategic 
environment  and  the  resulting  congressional  emphasis  on 
applied  research  short  of  production,  US  Army  Captain  Tod 
Jordan  examined  methods  for  applying  multicriteria  decision 
models  to  the  Army's  R  &  D  project  selection  process 
(Jordan,  1992) .  He  examines  the  AHP,  MAUT,  Goal 
Programming,  and  Graphical  Techniques,  presenting  a 
description  of  each  and  a  summary  of  their  strengths  and 
weaknesses . 

After  a  description  of  the  Army  R  &  D  Investment 
strategy,  the  author  describes  basic  decision  theory 
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concepts  and  presents  an  introduction  to  analytic  decision 
models.  He  then  describes  each  of  the  multicriteria 
decision  models,  again,  only  research  related  to  AHP  will  be 
described  here . 

In  his  description  of  AHP,  Jordan  states,  "This 
technique  is  quite  possibly  the  most  important  for 
quantifying  qualitative  expert  opinion  and  judgment  for 
decision  support,  at  least  in  terms  of  its  ease  of  use,  its 
parallel  to  actual  decision  making  processes,  and  its  broad 
applicability."  (Jordan,  1992).  Sentiments  similar  to 
these  were  also  expressed  by  the  previous  authors. 

Captain  Jordan  presents  an  example  of  AHP  that  has  for 
its  top-level  mandate  Enhance  Warfighting.  The  objectives 
that  make  up  the  next  level  of  the  hierarchy  are  Minimize 
Cost,  Minimize  Risk,  Maximize  Benefit,  Maximize  Capability, 
and  Maximize  Availability.  Under  this  level  would  be  the 
alternatives . 

From  this  exaimple  of  AHP  and  the  other  techniques 
described,  Jordan  concludes: 

For  the  Army  research  and  development  project 
selection  problem,  MCDM  hold  a  great  deal  of  promise. 
With  increased  pressure  from  Congress  for  the 
Department  of  Defense  to  assure  America's  technological 
defense  superiority  in  the  face  of  dramatically  reduced 
defense  spending,  there  will  be  increased  scrutiny  and 
accountability  for  the  Army's  R&D  resource  commitment 
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decisions.  Multicriteria  decision  models  offer  a  means 
to  develop  logically  supportable  solutions  to  this 
issue.  (Jordan,  1992) 
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III.  RESEARCH  METHODOLOGY 

As  Stated  previously,  the  purpose  of  this  study  is  to 
formulate  a  procedure  to  evaluate  and  prioritize  programs 
and  projects  in  the  resource  allocation  and  proposal 
evaluation  stages  of  the  procurement  process  in  an  Air  Force 
research  and  development  laboratory.  The  Analytical 
Hierarchy  Process  (AHP)  developed  by  Saaty  (Saaty,  1982) 
will  be  used  in  both  stages.  The  first  step  in  AHP  is  to 
determine  the  appropriate  attributes  and  subattributes  for 
both  stages  of  the  procurement  process.  The  next  step  in 
the  process  is  to  classify  the  attributes  into  functional 
hierarchies.  The  final  step  of  the  process  is  to  develop 
questionnaires  for  determining  the  relative  importance  of 
attributes  which  are  then  used  in  calculating  the  ranking 
and  weighting  of  the  attributes  and  subattributes. 

3.1.  Hierarchy  for  Projects  and  Proposals 

Functional  hierarchies  can  be  used  to  break  down 
complex  systems  into  their  constituent  parts  according  to 
their  essential  relationships  (Saaty,  1982) .  In  this  study, 
functional  hierarchies  were  developed  for  evaluating 
projects  before  the  Resource  Allocation  Committee  (RAC)  in 
the  resource  allocation  stage,  and  for  evaluating  contractor 
proposals  in  the  proposal  evaluation  stage.  For  the 
resource  allocation  hierarchy,  the  attributes  identified  are 
Need,  Technical  Approach,  Innovation,  Alternatives,  and 
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Additional.  The  attributes  identified  for  the  proposal 
evaluation  hierarchy  are  Understanding  the  Problem, 

Soundness  of  Approach,  Compliance  with  Requirements,  and 
Special  Technical  Factors. 

Because  a  hierarchy  represents  a  model  of  how  the 
brain  analyzes  complexity,  the  hierarchy  must  be 
flexible  enough  to  deal  with  that  complexity.  The 
levels  of  hierarchy  interconnect  like  layers  of  cell 
tissue  to  form  an  organic  whole  that  serves  a  certain 
function.  A  spiraling  effect  is  noticeable  when  we 
move  from  the  focus  expanding  the  hierarchy  to  the 
level  of  simple  elements  (Saaty,  1982) . 

To  further  define  the  main  attributes  in  the  resource 
allocation  and  proposal  evaluation  hierarchies,  a  second 
level  of  subattributes  was  added.  The  hierarchy  diagrams 
for  the  resource  allocation  and  proposal  evaluation  stages 
can  be  seen  in  Figure  3.1.1.  and  Figure  3.1.2.  respectively. 

In  this  study,  appropriate  attributes  were  established 
during  an  interview  with  the  Director  of  the  Electronic 
Warfare  Division.  These  attributes  have  been  used  in  the 
Division  for  several  years.  The  attributes  and 
subattributes  used  for  the  proposal  evaluation  stage  are 
similar  to  those  which  are  currently  used  for  evaluating  a 
research  and  development  effort  presently  under  contract. 
These  were  obtained  during  an  interview  with  a  division 
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project  engineer.  Definitions  for  all  attributes  and 
subattributes  is  in  contained  Appendix  A. 

To  see  how  to  view  the  hierarchies,  examine  Figure 
3.1.1,  the  hierarchy  for  resource  allocation.  On  the  first 
level,  one  of  the  attributes  is  Need.  The  subattributes 
under  Need  are  Generally  Critical,  Limited  Critical, 
Generally  Non- Critical ,  and  Limited  Non- Critical ,  these 
define  the  second  level  of  the  hierarchy.  The  final  level 
deals  with  the  project  alternatives  before  the  Resource 
Allocation  Committee  (RAC) . 

Hierarchy  for  Resource  Allocation 

First  level;  Need,  Technical  Approach,  Innovation, 
Alternatives,  and  Additional. 

Second  Level : 

Need:  Generally  Critical,  Limited  Critical, 

Generally  Non-Critical,  Limited  Non-Critical . 
Technical  Approach:  Project  Definition, 

Practicality,  Feasibility. 

Innovation:  High  Payoff  Approach,  Novel 

Approach,  Some  New  Ideas,  Routine. 

Alternatives :  None,  Others/This  Better,  Several 

Equal,  Many/Some  Better. 

Additional :  Follow-On  Program,  Joint  Program,  In- 

House,  Supports  Other  Programs,  Supports  Forecast  II. 
Third  Level :  Project  1,  Project  2,  Project  3 


FIGURE  3.1.2. 
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Hierarchy  for  Proposal  Evaluation 

First  Level :  Understanding  the  Problem,  Soundness  of 
Approach,  Compliance  with  Requirements,  Special  Technical 
Factors . 

Second  Level : 

Understanding  the  Problem:  Aircraft  Platform 
Requirements,  Array  Antenna  Configuration  Trade-Offs, 
Processing  and  Control,  Signal  Processing  and 
Switching,  Miscellaneous. 

Soundness  of  Approach;  Analysis,  Design,  Testing  and 
Demonstration  (T  &  D) . 

Compliance  with  Requirements;  Address  Minimum 
Analysis  Topics,  Meet  Minimum,  Design  Criteria,  Meet 
Minimum  T  &  D  requirements.  No  Restatement, 
Presentation. 

Special  Technical  Factors;  Design  of  Solid  State 
Arrays,  ECM  System  Development,  Digital  Controller 
Development,  Advanced  Processing  Investigation, 
Embedded  System  Software,  MMIC  Device  Application. 
Third  Level  (proposal  ratings) :  Exceptional,  Acceptable, 
Marginal,  and  Unacceptable. 

3.2.  Procedure  for  Data  Collection 

Recall,  one  of  the  objectives  of  this  study  was  to 
develop  an  easy  to  use  procedure  to  evaluate  and  prioritize 
projects  brought  before  the  RAC  and  proposals  sent  in  by 
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defense  contractors.  A  two  step  methodology  has  been 
developed  through  the  use  of  AHP  to  institute  a  procedure, 
recognizing  the  individual  differences  among  R  &  D 
organization's  resource  allocation  and  project  engineer's 
proposal  evaluation  priorities.  In  step  one,  data  on  the 
preferences  cimong  different  attributes  is  collected,  this  is 
done  using  pairwise  comparisons.  Pairwise  comparisons  are 
accomplished  by  comparing  two  attributes  within  a  given 
level  of  the  hierarchy  and  rating  them  with  respect  to  each 
other  on  a  scale  of  1  to  9 .  The  scale  reflects  the  degree 
of  importance  between  the  two  attributes.  The  relative 
weights  of  the  attributes  are  determined  in  the  second  step 
using  the  pairwise  comparison  data  of  step  one.  The  weights 
are  then  used  to  determine  the  scores  for  each  project  or 
proposal . 

In  his  development  of  AHP,  Saaty  states: 

Humans  have  the  ability  to  perceive  relationships 
among  the  things  they  observe,  to  compare  pairs  of 
similar  things  against  certain  criteria,  and  to 
discriminate  between  members  of  a  pair  by  judging  the 
intensity  of  their  preference  for  one  over  the  other. 
Then  they  synthesize  their  judgements  --  through 
imagination  or,  with  the  AHP,  through  a  new  logical 
process  -  -  and  gain  a  better  understanding  of  the  whole 
system.  (Saaty  1982) 


This  provides  the  basis  for  the  use  of  pairwise 
comparisons  in  determining  the  relative  importance  of  all 
the  attributes  and  subattributes  in  the  two  previously 


described  hierarchies. 

The  hierarchies  described  earlier  for  resource 
allocation  and  proposal  evaluation  were  used  to  develop 
questionnaires  for  evaluating  the  relative  importance  of  the 
attributes  and  subattributes.  For  example,  a  c[uestionnaire 
was  developed  to  ascertain  the  importance  of  Need  in  the 
resource  allocation  hierarchy.  The  respondent  was  given  a 
questionnaire  with  a  complete  set  of  instructions  for  its 
completion.  His  preferences  were  indicated  by  placing  a 
check  mark  under  the  appropriate  number  of  the  scale  on 
either  the  right  or  left  side  of  the  table,  depending  upon 
which  attribute  was  preferred.  A  copy  of  the  questionnaires 
and  their  instructions  is  included  in  Appendix  C. 

Basically,  the  questionnaire  requires  the  respondent  to 
determine  two  things:  1}  which  attribute/subattribute  is 
more  important,  and  2)  by  how  much. 

3.2.1.  Resource  Allocation  Stage 

To  show  how  this  process  can  be  applied  to  the  resource 
allocation  stage,  a  simulated  RAC  was  established. 

Personnel  included  on  this  committee  were  the  Division 
Chief /Director ,  Director  of  Analysis  Projects  for  the 
division,  and  three  project  engineers  who  have  extensive 
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experience  with  the  RAC.  Normally,  the  RAC  would  consist  of 
the  Division  Chief,  the  Division  Technical  Director,  and  all 
the  Branch  Chiefs. 

Each  of  these  individuals  was  given  a  questionnaire, 
complete  with  instructions,  and  asked  to  carefully  fill  out 
the  tables.  Their  responses  were  used  to  calculate  a  final 
score  for  each  project,  this  procedure  will  be  described  in 
detail  in  the  following  sections. 

3.2.2.  Proposal  Evaluation  Stage 

Application  of  the  process  to  the  proposal  evaluation 
stage  was  accomplished  by  examining  a  research  and 
development  effort  that  is  presently  under  contract.  The 
model  was  applied  using  the  actual  criteria  that  were  used 
to  evaluate  proposals  sent  in  response  to  this  effort. 
Because  of  the  security  classification  and  sensitive  nature 
of  some  of  the  proposals  sent  in  by  defense  contractors,  the 
project  engineer  asked  that  the  real  proposals  not  be  used 
in  this  evaluation. 

The  project  engineer  for  the  R  &  D  effort  used  in  this 
study  was  furnished  with  the  instructions  and  questionnaire 
for  evaluating  technical  proposals  found  in  Appendix  D.  His 
responses  were  used  to  evaluate  and  prioritize  three 
fictitious  proposals.  Again,  this  procedure  will  be 
described  in  detail  in  the  following  sections. 
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3.3.  Calculation  of  Priority  Weights 

The  procedure  for  calculating  the  priority  weights  is 
the  same  for  both  the  resource  allocation  and  proposal 
evaluation  stages  in  this  study.  Different  weights  are 
assigned  to  the  attributes  by  translating  the  data  from  a 
given  comparison  table  to  a  preference  matrix.  The  eigen 
values  of  the  preference  matrix  then  become  the  weights  for 
each  attribute. 

An  approximate  method  is  used  to  determine  the  eigen 
values  of  the  preference  matrix.  With  the  approximate 
method,  the  weights  are  calculated  by  summing  each  column  of 
the  preference  matrix  individually  and  dividing  each  element 
of  the  column  by  the  sum.  Next  find  the  mean  for  each  row 
of  the  matrix,  this  becomes  the  weight  (approximate  eigen 
value)  for  that  row.  A  computer  spreadsheet  has  been 
developed  to  calculate  the  approximate  eigen  values. 

The  above  procedure  is  best  illustrated  through  an 
example.  The  proposal  evaluation  comparison  Table  3.3.1  was 
completed  by  a  project  engineer  in  the  division  studied. 

Previously  it  was  shown  that  there  are  four  main 
attributes  in  the  proposal  evaluation  hierarchy,  the  initial 
step  is  the  formation  of  a  four-by-four  matrix;  this  is 
known  as  the  preference  matrix,  and  in  this  case,  has  16 
entries.  Notice  that  the  diagonal  of  the  matrix  consists  of 
all  Is,  this  is  because  each  criteria  is  compared  with 
itself.  In  translating  the  comparisons  table  to  a  matrix. 
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first  determine  whether  the  row  is  more  important  or  the 
column  is  more  important.  With  respect  to  proposal 
evaluation,  if  the  row  is  more  important,  the  value  from  the 
comparisons  table  is  copied  into  this  entry.  Conversely,  if 
the  column  is  more  important,  the  reciprocal  of  the  value 
from  the  comparisons  table  is  copied  into  this  entry.  A 
preference  matrix  based  on  the  Table  3.3.1  (the  comparisons 
table  for  proposal  evaluation)  is  shown  in  Table  3.3.2. 

TABLE  3.3.1. 
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TABLE  3.3.3. 


Priority 

Weight 

Understanding  the 
Problem 

0.54 

Soundness  of 

Approach 

0.29 

Compliance  with 
Requirements 

0.07 

Special  Technical 
Factors 

0.10 

Total 

1.00 

3.4«  Calculation  of  Project /Proposal  Scores 

After  the  pairwise  comparisons  are  completed  at  all 
levels  of  the  hierarchies  and  the  weights  established  from 
the  preference  matrices,  a  single  score  of  the  overall 
performance  of  each  alternative  can  be  calculated. 

3.4.1.  Resource  Allocation  Project  Scores 

The  final  ranking  of  projects  brought  before  the  RAC  is 
derived  by  using  a  combination  of  methods  from  the  existing 
process  and  methods  used  in  AHP. 

Again,  a  brief  example  will  be  useful  in  illustrating 
the  methodology  for  calculating  f-'ie  project  score.  Figure 
3.4.1.  shows  one  level  of  the  resource  allocation  hierarchy 
along  with  the  weights  calculated  using  the  method  of  the 
previous  section.  The  score  for  an  alternative  is 
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determined  by  the  sum  of  products  of  the  value  of  the 
attribute  and  its  weight.  The  interim  score  calculated  in 
this  example  is  only  for  Technical  Approach,  similarly 
scores  would  be  calculated  for  the  other  attributes  at  level 
1  (Need,  Innovation,  Alternatives,  and  Additional) .  The 
interim  scores  would  then  be  summed  across  the  hierarchy  for 
all  alternatives  to  determine  their  final  scores. 

FIGURE  3.4.1. 


.281) 


Project  1  =  .286[.081{.090)+.638(.066)+.281(.549)] 

=  .058 

similarly  for  Project  2,  we  have 

Project  2  =  .286  [  .081 ( .303) +  .638  ( .363) +.281 ( .120) ] 

=  .100 

finally,  for  Project  3,  we  have 

Project  3  =  . 286  [ . 081 { . 607) + . 638  { . 571) + . 281 ( . 120 ) ] 


.  128 
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This  would  provide  an  interim  score  at  Technical 
Approach,  similar  interim  scores  would  be  calculated  at  the 
other  level  one  attributes  and  summed  to  determine  the  final 
score  for  each  project. 

Final  Scorepj,Qjgj,t.  ^  =  Score  at  Need  +  Score  at  Technical  of  Approach  + 

Score  at  Innovation  +  Score  at  Alternatives  + 
Score  at  Additional 

This  process  is  called  "folding  back  the  tree." 

It  was  noted  in  Section  Two  that  in  the  resource 
allocation  process,  several  committee  members  evaluate  the 
projects  and  their  individual  responses  must  be  aggregated. 
To  accomplish  this  aggregation  in  this  study,  a  method 
similar  to  that  used  in  the  existing  process  will  be 
applied. 

Final  scores  for  each  project  will  be  calculated  for 
each  evaluator.  Based  on  the  final  scores,  the  projects 
will  be  rank  ordered  for  each  committee  member,  for  example, 
the  committee  chairman  may  have  the  projects  ranked  as 
follows:  project  three  ranked  one,  project  one  ranked  two, 
and  project  two  ranked  three.  The  interim  rankings  for  each 
evaluator  would  then  be  averaged  to  determine  the  final 
rankings . 
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3.4.2.  Proposal  Evaluation  Scores 

Because  procurement  regulations  prohibit  the  direct 
comparison  of  contractors  on  a  numerical  scale  (APR  70-30, 
par  30d.),  we  cannot  use  pairwise  comparisons  to  compare  the 
contractors.  Thus  an  absolute  method  of  evaluating  the 
proposals  needed  to  be  adopted  in  the  model . 

The  absolute  ratings  used  to  evaluate  the  proposals  are 
exceptional,  acceptable,  marginal,  and  unacceptable.  These 
are  the  ratings  specified  in  the  regulation  governing 
technical  evaluations  of  proposals  in  the  organization 
studied  (WRDC  Regulation  70-5) .  Each  rating  could  have  a 
plus  (+)  or  minus  (-)  associated  with  it,  i.e.,  a  rating  of 
'exceptional  +'  is  possible. 

The  method  applied  here  is  based  on  the  one  proposed  by 
Francis  and  Stonebraker.  In  their  application  of  AHP  to 
source  selection,  they  used  color  coded  ratings  which  would 
correspond  to  the  four  ratings  used  in  this  study. 

For  each  criterion  at  the  lowest  level  of  the 
hierarchy,  pairwise  comparisons  were  initially  employed 
to  determine  the  relative  importance  of  the  colors  with 
respect  to  each  of  these  criteria.  This  permits  item 
specific  meanings  for  the  color  ratings.  (Francis  and 
Stonebraker,  1991) 

To  visualize  what  they  mean  by  item  specific  meanings 
for  the  ratings,  consider  this  example,  the  relative 
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importance  of  unacceptable  and  exceptional  will  probably  be 
different  when  rating  Aircraft  Platform  Requirements  versus 
Presentation. 

The  last  section  of  the  proposal  evaluation 
questionnaire  contains  comparison  tables  for  the  four 
ratings  considering  the  lowest  level  criteria  in  the 
proposal  evaluation  hierarchy.  Also  contained  in  this 
section  are  tables  to  evaluate  each  proposal  independently 
against  the  criteria  (see  Appendix  C) .  Notice  that  each  of 
the  three  proposals  could  possibly  receive  the  same  rating. 

The  final  score  the  proposal  receives  is  calculated  as 
follows:  the  rating  the  proposal  receives  for  a  given 

criterion  is  assigned  the  weight  for  that  rating  (weights 
determined  from  the  pairwise  comparisons) ,  this  is  done  for 
all  the  criteria  in  the  lowest  level  of  the  hierarchy;  the 
remainder  of  the  calculation  is  the  same  "folding  back  the 
tree"  process  found  in  the  previous  section. 

Here,  a  brief  example  will  be  useful  in  illustrating 
the  methodology  for  calculating  the  final  score.  Figure 
3.4.2.  shows  one  level  of  the  proposal  evaluation  hierarchy 
along  with  the  weights  calculated  using  the  method  of  the 
Section  3.3.  Assume  the  proposal  being  evaluated  was  rated 
exceptional  for  Analysis  Task,  and  acceptable  for  both 
Design  Task  and  Testing  and  Demonstration  Task.  The  interim 
score  at  Soundness  of  Approach  would  be  0.150.  The  final 
score  for  the  project  would  be  the  sum  of  the  interim  scores 


at  Understanding  the  Problem,  Soundness  of  Approach, 
Compliance  with  Requirements,  and  Special  Technical  Factors. 


FIGURE  3.4.2. 


Interim  score  at  Soundness  of  Approach  =  . 290 [ . 595 ( . 691) + 

.289 (.284)+.246(.096) ]  =  .150 

To  be  consistent  with  present  procurement  procedures, 
the  final  score  must  be  translated  to  an  overall  rating  of 
exceptional,  acceptable,  marginal,  or  unacceptable. 

Again,  Francis  and  Stonebraker  provide  guidance  for 
making  this  translation: 
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In  our  application  to  the  case  study,  we  investigated  ways  to 
translate  the  numerical  "composite  priorities"  provided  by  the  AHP 
back  into  a  traditional  color  rating.  This  makes  the  application 
of  AHP  to  source  selection  consistent  with  established  procedures. 
Moreover,  the  AHP  methodology  is  almost  totally  "hidden"  from  the 
users,  giving  the  appearance  that  they  are  "conducting  business  as 
usual."  It  should  be  emphasized  that 

1.  all  of  the  advantages  of  applying  the  AHP  are 
retained, 

2.  source  selection  proceeds  as  specified  in 
existing  regulations,  directives  and  policies,  and 

3.  evaluators  and  decision  makers  are  not 
confronted  with  changing  to  and  learning  a 

dramatically  different  method.  (Francis  and  Stonebraker,  1991) 

The  actual  translation  of  the  final  scores  into  the 
ratings  they  prescribed  is  done  as  follows  : 

1.  For  the  attributes  being  evaluated,  set  all  evaluations 
to  exceptional.  Note  the  resultant  final  score.  This 
becomes  the  upper  limit  for  exceptional. 

2.  Set  all  evaluations  to  acceptable.  Note  the  resultant 
final  score. 

3.  The  average  of  the  final  scores  obtained  in  steps  1  and 
2  is  the  lower  limit  of  the  exceptional  range  and  the  upper 
limit  of  the  acceptable  range. 

4.  Repeat  steps  1  through  3  for  acceptable  and  marginal. 
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The  following  table  provides  hypothetical  results  using  the 
above  procedure. 


TABLE  3.4.1. 


Evaluations 

Resultant 

Rating 

all  set  to: 

final  score 

Average 

Code 

Range 

Exceptional 

0.54 

0.430 

0.430 

-  0.540 

Acceptable 

0.32 

0.235 

0.235 

-  0.429 

Marginal 

0.15 

0.095 

0.095 

-  0.234 

Unacceptable 

0.04 

The  extreme  points  for  each  rating  range  could  then  be 
used  for  the  plus  and  minus  designations.  Assume  proposal 
one  receives  a  final  score  of  0.440,  thus  its  rating  would 
then  be  exceptional. 
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IV.  PERFORMANCE  EVALUATION 


The  model  and  application  procedures  developed  in  this 
study  were  used  in  two  stages  of  the  procurement  process  in 
an  Air  Force  research  and  development  laboratory:  the 
resource  allocation  stage  and  the  proposal  evaluation  stage. 
In  the  resource  allocation  stage,  the  method  for 
prioritizing  projects  was  briefed  (and  questionnaires 
provided)  to  five  members  of  the  division  studied,  these 
individuals  comprised  the  Resource  Allocation  Committee 
(RAC)  used  in  the  study.  Typically,  the  RAC  would  be  made 
up  of  the  Division  Chief,  Division  Technical  Director,  and 
the  four  Branch  Chiefs.  The  results  provided  by  the  RAC 
members  will  be  analyzed  and  the  projects  ranked  according 
to  the  procedure  outlined  in  the  previous  section.  To 
demonstrate  how  the  method  can  be  applied  in  the  proposal 
evaluation  stage,  a  project  engineer  in  the  division  was 
asked  to  complete  the  questionnaire.  The  criteria/ 
attributes  used  in  both  stages  of  the  study  were  provided  by 
the  division  and  have  been  previously  used  in  practice. 
Actual  names  and  addresses  of  the  respondents  in  this  study 
have  been  withheld  for  confidentiality  reasons. 

With  the  resource  allocation  stage  of  the  study,  a 
pairwise  comparison  of  the  attributes  and  the  projects  in 
the  resource  allocation  hierarchy  was  completed  by  each 
member  of  the  RAC  using  the  questionnaire  in  Appendix  C. 
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Similarly,  the  project  engineer  completed  a  pairwise 
comparison  of  the  attributes  and  ratings  for  proposals  in 
the  project  evaluation  hierarchy  using  the  questionnaire  in 
Appendix  D.  The  proposals  were  then  evaluated  against  the 
lowest  level  criteria  in  the  proposal  evaluation  hierarchy 
using  the  evaluation  tables  in  the  last  part  of  Appendix  D. 

The  data  were  analyzed  by  the  procedure  outlined  in 
Section  Three.  The  results  of  both  of  these  analyses  are 
fully  discussed  in  the  following  sections . 

4.1.1.  Resource  Allocation  Stage 

There  were  five  people  from  the  division  who  completed 
questionnaires  for  this  stage  of  the  process.  Each 
respondent's  results  will  be  analyzed  separately  and 
summarized  in  the  following  pages. 

Recall,  the  attributes  identified  at  the  first  level  of 
the  resource  allocation  hierarchy  are:  Need,  Technical 
Approach,  Innovation,  Alternatives,  and  Additional.  The 
preference  matrices  for  each  evaluator  considering  these 
level -one  attributes  is  presented  in  Table  4.1.1.  The 
priority  weights  (PW)  shown  are  the  approximate  eigenvalues 
of  the  comparison  matrix. 

Looking  at  Table  4.1.1,  it  is  seen  that  all  evaluators 
agree  that  Need  is  the  most  important  attribute.  This  is  to 
be  expected,  why  spend  scarce  research  and  development 
dollars  on  something  that  is  not  needed.  Moreover,  all 


evaluators,  except  evaluator  five,  agree  that  Technical 
Approach  is  second,  and  Additional  is  third. 


TABLE  4.1.1. 


Project  Evaluation 

Need 

TA 

ADD 

ALT 

IN 

PW 

Need 

1 

2 

4 

4 

4 

.412 

Technical  Approach  (TA) 

■B 

1 

5 

4 

3 

.308 

Additional  (ADD) 

1/4 

1/5 

1 

1 

1/2 

.076 

Alternatives  (ALT) 

1/4 

1/4 

1 

1 

1/2 

.079 

Innovation  (IN) 

1/4 

1/3 

2 

2 

1 

.126 

Project  Evaluation 

Need 

TA 

ADD 

ALT 

IN 

PW 

Need 

1 

1 

5 

5 

3 

.358 

Technical  Approach  (TA) 

1 

1 

5 

5 

2 

.329 

Additional  (ADD) 

1/5 

1/5 

1 

1 

1/2 

.087 

Alternatives  (ALT) 

1/5 

1/5 

1 

1 

1/2 

.087 

Innovation  (IN) 

1/3 

1/2 

2 

2 

1 

.140 

Project  Evaluation 

Need 

TA 

ADD 

ALT 

IN 

PW 

Need 

1 

2 

7 

5 

2 

.430 

Technical  Approach  (TA) 

1/2 

1 

5 

2 

2 

.253 

Additional  (ADD) 

1/7 

1/5 

1 

1/2 

1/2 

.059 

Alternatives  (ALT) 

1/5 

1/2 

2 

1 

1 

.  116 

Innovation  (IN) 

1/2 

1/2 

2 

1 

1 

.  142 

Project  Evaluation 

Need 

TA 

ADD 

ALT 

IN 

PW 

Need 

1 

5 

3 

3 

S 

.400 

Technical  Approach  (TA) 

1/5 

1 

7 

5 

6 

.286 

Additional  (ADD) 

1/3 

1/7 

1 

5 

1/5 

.105 

Alternatives  (ALT) 

1/3 

1/5 

1/5 

1 

1/5 

.055 

Innovation  (IN) 

1/5 

1/6 

5 

5 

1 

.155 

Project  Evaluation 

Need 

TA 

ADD 

ALT 

IN 

PW 

Need 

1 

1 

2 

3 

4 

.323 

Technical  Approach  (TA) 

1 

1 

2 

4 

3 

.326 

Additional  (ADD) 

1/2 

1/2 

1 

1 

1 

.133 

Alternatives  (ALT) 

1/3 

1/4 

1 

1 

2 

.124 

Innovat ion  (IN) 

1/4 

1/3 

1 

1/2 

1 

.095 

For  brevity,  preference  matrices  for  the  second  and 
third  levels  of  the  resource  allocation  hierarchy  for  each 
evaluator  are  not  included  here;  however,  the  results  of  the 
analysis  of  the  preference  matrices  for  each  evaluator  are 
summarized  in  the  tables  that  follow. 

The  summary  of  the  comparison  of  subattributes  under 
Need:  Generally  Critical,  Limited  Critical,  Generally  Non- 
Critical,  and  Limited  Non-Critical  is  shown  in  Table  4.1.2. 
The  elements  in  the  table  are  the  priority  weights 


calculated  for  each  evaluator. 


The  second- level  attributes  identified  to  further 
define  Technical  Approach  are;  Project  definition, 
Practicality  and  Feasibility.  The  priority  weights 
calculated  for  each  of  the  five  evaluators  are  shown  below 
in  Table  4.1.3. 


Evaluator 

TECHNICAL  APPROACH 

1 

2 

3 

4 

5 

Project  Definition 

.321 

.259 

.525 

.081 

.400 

Practicality 

.225 

.171 

.142 

.638 

.400 

Feasibility 

.454 

.570 

.334 

.281 

.200 

High  Payoff  Approach,  Novel  Approach,  Some  New  Ideas, 
and  Routine  are  the  second- level  subattributes  under 
Innovation.  The  priority  weights  calculated  for  each  of  the 


five  evaluators  are  shown  in  below  in  Table  4.1.4. 
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TABLE  4.1.4. 


Evaluator 

INNOVATION 

1 

2 

3 

4 

5 

High  Payoff  Approach 

.601 

.618 

.529 

.684 

.454 

Novel  i^proach 

.235 

.246 

.244 

.164 

.191 

Some  New  Ideas 

.114 

.091 

.183 

.110 

.191 

Routine 

.050 

.045 

.045 

.042 

.163 

The  subattributes  identified  under  the  level -one 
attribute  Alternatives  are:  None,  Others/This  Better, 
Several  Equal,  Many/Some  Better.  The  priority  weights 
calculated  for  each  evaluator  are  shown  below  in  Table 
4.1.5. 


TABLE  4.1.5. 


Evaluator 

ALTERNATIVES 

1 

2 

3 

4 

5 

None 

.641 

.571 

.625 

.675 

.591 

Others/This  Better 

.198 

.228 

.234 

.105 

.245 

Several  Equal 

.096 

.127 

.106 

.115 

.064 

Many/Some  Better 

.064 

.073 

.035 

.105 

.101 

The  final  level -one  attribute  is  Additional,  the 
subattributes  under  this  are:  Follow-On,  Joint  Program,  In- 
House,  Supports  Others,  Supports  Forecast  II.  The  priority 


weights  calculated  for  each  evaluator  are  shown  in  Table 

4.1.6. 
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Once  all  of  the  priority  weights  are  calculated  for  all 
of  the  attributes  and  subattributes,  the  weights  the 
proposals  receive  considering  each  of  the  subattributes  need 
to  be  calculated  for  each  evaluator.  The  results  of  this 
analysis  are  summarized  in  Table  4.1.7. 

The  resource  allocation  hierarchies  complete  with 
weights  for  all  evaluators  are  presented  in  Figures  4.1.1. 
through  4.1.5.  respectively. 


TABLE  4.1.6. 


Evaluator 

ADDITIONAL 

1 

2 

3 

4 

5 

Follow-On 

.222 

.337 

.489 

.052 

.169 

Joint  Program 

.267 

.203 

.122 

.488 

.144 

In-House 

.144 

.112 

.243 

.137 

.144 

Supports  Others 

.244 

.269 

.094 

.201 

.255 

Supports  Forecast  II 

.122 

.079 

.051 

.122 

.288 
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TABLE  4.1.7. 


Evaluator 


SUBATTRIBUTE 

PROPOSAL 

1 

2 

3 

4 

5 

1 

.701 

.467 

.056 

.065 

.263 

Generally  Critical 

2 

.234 

.467 

.242 

.259 

.288 

3 

.065 

.067 

.702 

.675 

.448 

1 

.703 

.455 

.056 

.067 

.198 

Limited  Critical 

2 

.222 

.455 

.242 

.467 

.490 

3 

.075 

.090 

.702 

.467 

.312 

1 

.525 

.429 

.056 

.315 

.333 

Generally  Non-Critical 

2 

.334 

.429 

.263 

.204 

.333 

3 

.142 

.143 

.681 

.481 

.333 

1 

.765 

.429 

.055 

.714 

.333 

Limited  Non-Critical 

2 

.197 

.429 

.227 

.143 

.333 

3 

.070 

.143 

.717 

.143 

.333 

1 

.725 

.122 

.078 

.090 

.233 

Project  Definition 

2 

.200 

.230 

.234 

.303 

.367 

3 

.075 

.648 

.688 

.607 

.400 

1 

.717 

.633 

.333 

.066 

-333 

Practicality 

2 

.218 

.260 

.333 

.363 

.333 

3 

.065 

.106 

.333 

.571 

.333 

1 

.717 

.455 

.117 

.549 

.264 

Feasibility 

2 

.218 

.455 

.538 

.331 

.329 

3 

.065 

.090 

.345 

.120 

.407 
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TABLE  4 ♦ 1 . 7 .  Continued 


Evaluator 

SOBATTRIBUTE 

PROPOSAL 

1 

2 

3 

4 

5 

1 

.668 

.648 

.052 

.071 

.218 

High  Pay-off 

2 

.257 

.230 

.632 

.206 

.348 

3 

.075 

.122 

.316 

.723 

.434 

1 

.607 

.778 

.052 

.140 

.172 

Novel  i^proach 

2 

.303 

.111 

.579 

.574 

.350 

3 

.090 

.111 

.368 

.286 

.478 

1 

.638 

.600 

.052 

.140 

.225 

Some  New  Ideas 

2 

.273 

.200 

.579 

.286 

.454 

3 

.089 

.200 

.368 

.574 

.321 

1 

.694 

.333 

.808 

.090 

.537 

Routine 

2 

.231 

.333 

.074 

.303 

.268 

3 

.075 

.333 

.118 

.607 

.195 

1 

.600 

.600 

.061 

.333 

.333 

None 

2 

.200 

.200 

.723 

.333 

.333 

3 

.200 

.200 

.216 

.333 

.333 

1 

.600 

.600 

.052 

.333 

.407 

Others/This  Better 

2 

.200 

.200 

.685 

.333 

.329 

3 

.200 

.200 

.263 

.333 

.264 

1 

.537 

.537 

.090 

.138 

.333 

Several  Equal 

2 

.268 

.268 

.556 

.239 

.333 

! 

j 

3 

.  195 

.195 

.354 

.623 

.333 

TABLE  4.1.7.  Continued 


Evaluator 

SUBATTRIBUTE 

PROPOSAL 

1 

2 

3 

4 

5 

I 

.333 

.333 

.667 

.140 

.524 

Many/Scane  Better 

2 

.333 

.333 

.167 

.286 

.304 

3 

.333 

.333 

.167 

.574 

.172 

1 

.590 

.655 

.750 

.600 

.411 

Follow-On  Program 

2 

.145 

.158 

.125 

.200 

.328 

3 

.266 

.187 

.125 

.200 

.261 

1 

.443 

.443 

.333 

.333 

.333 

Joint  Program 

2 

.387 

.387 

.333 

.333 

.333 

3 

.170 

.170 

.333 

.333 

.333 

1 

.333 

.333 

.333 

.686 

.417 

In-House 

2 

.333 

.333 

.333 

.211 

.231 

3 

.333 

.333 

.333 

.102 

.352 

1 

.333 

.333 

.333 

.089 

.235 

Supports  Others 

2 

.333 

.333 

.333 

.658 

.325 

3 

.333 

.333 

.333 

.253 

.440 

1 

.333 

.333 

.333 

.138 

.250 

Supports  FORECAST  II 

2 

.333 

.333 

.333 

.239 

.500 

3 


.333 


.333 


.333 


.623 
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[GORE  4.1.1 


52 


FIGURE  4.1.2 
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FIGURE  4.1.5 
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With  the  hierarchies  for  each  evaluator  completed,  the 
projects  can  be  ranked  (for  each  evaluator)  according  to  the 
final  score  calculated  for  each  project  using  the  method 
outlined  in  Section  Three.  Table  4.1.8  provides  the  final 
score  given  each  project  by  each  evaluator,  and  also  the 
rank  order  of  the  projects  for  a  given  evaluator.  For 
example,  if  the  rank  order  is  3  2  1,  this  means  the  final 

scores  indicate  Project  1  is  ranked  3,  Project  2  is  ranked 
2,  and  Project  3  is  ranked  1  for  the  given  evaluator. 


TABLE  4.1.8. 


Project 

Evaluator 

1 

2 

3 

Rcuik  Order 

1 

.651 

.247 

.111 

1 

2 

3 

2 

.468 

.361 

.173 

1 

2 

3 

3 

.147 

.340 

.515 

3 

2 

1 

4 

.191 

.320 

.490 

3 

2 

1 

5 

.287 

.346 

.339 

3 

1 

2 

Recall  from  Section  Three,  the  overall  ranking  of  the 
projects  is  obtained  by  averaging  the  rankings  obtained  for 
each  evaluator.  Using  the  results  found  in  Table  4.1.8,  the 
overall  ranking  for  the  three  proposals  was  calculated  and 
the  results  presented  in  Table  4.1.9. 
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TABLE  4.1.9. 


Evaluator 

Overall 

Rank 

Project 

1 

2 

3 

4 

5 

2 

2 

2 

2 

2 

1 

1.8 

3 

3 

3 

1 

1 

2 

2.0 

1 

1 

1 

3 

3 

3 

2.2 

The  results  of  this  analysis  show  that  Project  Two  has 
the  lowest  'overall  rank',  thus  it  is  the  top  ranked 
project.  Project  Three  is  the  next  highest  ranked,  and 
Project  One  is  the  lowest  ranked  according  to  analysis  of 
the  evaluators  questionnaires. 

In  this  study,  three  projects  were  used  to  illustrate 
how  AHP  can  be  use  to  rank  projects  brought  before  the  RAC. 
In  an  actual  application  of  the  procedure,  there  will  likely 
be  several  more  projects  that  need  to  be  evaluated;  however, 
the  same  simple  steps  will  be  followed  to  determine  their 
overall  ranking.  With  the  overall  ranking  and  knowledge  of 
how  much  money  is  available  to  fund  new  efforts,  a  funding 
line  can  be  drawn  with  all  projects  above  the  line  being 


funded. 


A  project  engineer  from  the  division  was  recruited  to 
complete  the  questionnaire  for  this  stage  of  the  process. 

In  a  meeting  held  in  August  1992,  this  individual  provided 
the  criteria/attributes  that  were  used  in  this  study.  As 
previously  stated,  this  is  the  same  criteria  that  was 
actually  used  to  evaluate  proposals  sent  in  response  to  a 
request  for  proposal  prepared  by  the  project  engineer. 

The  results  of  the  analysis  of  the  project  engineer's 
responses  is  contained  in  the  following  pages.  As  with  the 
previous  stage,  the  priority  weights  in  the  following  tables 
were  calculated  as  prescribed  in  Section  Three. 

The  attributes  identified  at  the  first  level  of  the 
proposal  evaluation  hierarchy  are:  Understanding  the 
Problem,  Soundness  of  Approach,  Compliance  with 
Requirements,  and  Special  Technical  Factors.  The  preference 
matrix  for  the  level -one  attributes  is  presented  in  Table 
4.2,1. 

Examining  this  preference  matrix,  it  can  be  noted  that 
Understanding  the  Problem  is  the  highest  rated  attribute, 
next  is  Soundness  of  Approach,  followed  by  Special  Technical 
Factors,  and  finally  Compliance  with  Requirements.  These 
weights  are  as  anticipated;  because  the  contractor  is 
expected  to  comply  with  the  requirements  set  forth  in  the 
request  for  proposal,  it  follows  that  Compliance  with 
Requirements  should  be  the  lowest  rated  attribute. 


59 


TABLB  4.2.1. 


PROPOSAL  EVALUATION 

UP 

SA 

CR 

STF 

PW 

Understanding  the 
Problem  (UP) 

1 

3 

6 

5 

.542 

Soundness  of 
;^proach  (SA) 

1/3 

1 

5 

4 

.290 

Compliance  with 
Requirements  (CR) 

1/6 

1/5 

1 

1/2 

.066 

Special  Technical 
Factors  (STF) 

1/5 

1/4 

2 

1 

.103 

The  preference  matrix  for  the  subattributes  Aircraft 
Requirements,  Array  Configuration,  Digital  Control,  RF 
Processing,  and  Miscellaneous  found  under  Understanding  the 
Problem  is  shown  in  Table  4.2.2. 


TABLE  4.2.2. 


UNDERSTANDING  (UP) 

AR 

AC 

DC 

RF 

MC 

PW 

Aircraft  Requirements  (AR) 

m 

1/2 

1/2 

1/2 

1 

.122 

Array  Configuration  (AC) 

2 

1 

1 

2 

3 

.297 

Digital  Control  (DC) 

2 

1 

m 

2 

3 

.297 

RF  Processing  (RF) 

2 

1/2 

1/2 

1 

2 

.183 

Miscellaneous  (MC) 

1 

1/3 

1/3 

1/2 

1 

.  102 

I 


Looking  at  Table  4.2.2,  Array  Configuration  and  Digital 
Control  are  of  equal  importance,  followed  by  RF  Processing, 
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Aircraft  requirements,  and  Miscellaneous,  respectively. 

This  is  to  be  expected  considering  the  criteria  applies  to 
the  development  of  an  electronic  countermeasures  antenna 
controller. 

Table  4.2.3  shows  the  preference  matrix  for  the 
subattributes  found  under  Soundness  of  Approach,  namely, 
Analysis  Task,  Design  Task,  and  Testing  and  Demonstration  (T 
&  D)  task. 


TABLE  4.2.3. 


SOUNDNESS  OF  APPROACH 

AT 

DT 

TD 

PW 

Analysis  Task  (AT) 

1 

3 

5 

.619 

Design  Task  (DT) 

1/3 

1 

4 

.284 

T  &  D  Task  (TD) 

1/5 

1/4 

1 

.096 

As  indicated  in  Table  4.2.3,  Analysis  task  is 
considered  the  most  important  attribute  under  Soundness  of 
Approach,  followed  by  Design  Task  and  T  &  D  Task. 

The  third  attribute  in  the  first  level  of  the  proposal 
evaluation  hierarchy  is  Compliance  with  Requirements.  Its 
subattributes  are  Analysis  Topics,  Design  Criteria,  T  &  D 
Requirements,  Technical  Discussion,  and  Presentation.  The 
preference  matrix  can  be  seen  in  Table  4.2.4. 


As  expected.  Presentation  is  not  as  important  as  the 
other  subattributes  under  Compliance  with  Requirements.  The 
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contractor  is  expected  to  submit  a  neat,  grammatically 
correct  proposal.  Consistent  with  the  weights  for  the 
previous  attribute,  Analysis  Topics  is  again  considered  more 
important  than  Design  Criteria,  followed  by  T  &  D 
Requirements  and  Technical  Discussion. 


TABLE  4.2.4. 


COMPLIANCE 

AT 

DC 

TR 

TD 

PR 

PW 

Analysis  Topics  (AT) 

1 

3 

5 

3 

7 

.456 

Design  Criteria  (DC) 

1/3 

1 

3 

2 

6 

.235 

T  &  D 

Requirements  (TR) 

1/5 

1/3 

1 

2 

6 

.154 

Technical 

Discussion  (TD) 

1/3 

1/2 

1/2 

1 

3 

.114 

Presentation  (PR) 

1/7 

1/6 

1/6 

1/3 

1 

.041 

The  final  set  of  subattributes  in  the  hierarchy  is 
under  Special  Technical  Factors,  these  being  Solid  State 
Array  Design,  ECM  System  Development,  Digital  Control, 
Advanced  Processing,  Embedded  System  Software  and  Algorithm 
(ESS  &  A)  Development,  and  MMIC  Application.  The  preference 
matrix  for  these  subattributes  is  displayed  in  Table  4.2.5. 

Examining  the  priority  weights  in  Table  4.2.5,  it  is 
noted  that  the  weights  for  these  subattributes  consistent 
with  what  can  be  expected  for  an  antenna  controller 
development  effort.  Solid  State  Array  Design  carries  the 


most  weight,  followed  by  MMIC  Application,  Digital  Control 
and  software  development  considerations. 
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TABLE  4.2.5 


SPECIAL  TECHNICAL 
FACTORS 

SS 

ECM 

DC 

AP 

ESS 

MA 

PW 

Solid  State  Array 
Design  (SS) 

1 

2 

2 

2 

2 

2 

.291 

ECM  System 
Development  (ECM) 

1/2 

1 

1/2 

1/2 

2 

1/2 

.122 

Digital 

Control  (DC) 

1/2 

2 

1 

1 

1 

1 

.151 

Advanced 
Processing  (AP) 

1/2 

2 

1 

1 

1/2 

1/2 

.128 

ESS  &  A 

Development  (ESS) 

1/2 

1/2 

1 

2 

1 

1 

.145 

MMIC 

implication  (MA) 

1/3 

2 

1 

2 

1 

1 

.163 

The  next  thing  to  be  done  in  this  stage  is  to  determine 
the  item  specific  weights  for  the  proposal  ratings 
Exceptional,  Acceptable,  Marginal,  and  Unacceptable. 

The  preference  matrix  in  Table  4.2.6  illustrates  the 
method  for  arriving  at  priority  weights  for  the  proposal 
ratings  when  considering  the  second  level  subattributes. 
Immediately  following  Table  4.2.6  in  Table  4.2.7  is  a 
summary  of  the  remaining  preference  matrices  for  the  item 
specific  proposal  ratings.  The  entries  in  the  table  under 
the  four  ratings  (Exceptional,  Acceptable,  Marginal,  and 


Unacceptable)  are  the  priority  weights  for  the  given 
subattributes . 


TABLE  4.2.6. 


AIRCRAFT  REQUIREMENTS 


Exceptional  (EX) 


Acceptcible  (AC) 


Marginal  (MG) 


Unacceptcible  (UA) 


TABLE  4.2.7 


SUBATTRIBUTE 


Aircraft  Requirements 


Array  Configuration 


Digital  Control 


RF  Processing 


Miscellcineous 


Analysis  Task 


Design  Task 


T  &  D  Task 


Analysis  Topics 


Design  Criteria 


T  &  D  Requirements 


Technical  Discussion 


Proposal  Rating 

EX 

AC 

MG 

UA 

.485 

.344 

.126 

.045 

.575 

.281 

.103 

.039 

.526 

.324 

.103 

.049 

.557 

.293 

.097 

.040 

.527 

.325 

.110 

.042 

.595 

.260 

.101 

.040 

.565 

.289 

.108 

.042 

.575 

.246  1 

.  120 

.064 

.584 

.241 

.121 

.051 

.556 

.306 

.093 

.043 

.513 

.305 

.134 

.052 

.598 

.264 

.  091 

.  043 

TABLE  4.2.7.  Continued 


SUBATTRIBDTE 


Presentation 


SS  Array  Design 


ECM  System  Development 


Digital  Controller  Dev 


Advanced  Processing  Inv 


ESS  &  A  Development 


MMIC  Device  Application 


Proposal  Rating 

EX 

AC 

MG 

UA 

.510 

.292 

.146 

.057 

.547 

.272 

.119 

.054 

.578 

.254 

.118 

.045 

.507 

.328 

.120 

.046 

.577 

.281 

.096 

.047 

.521 

.326 

.099 

.056 

.584 

.258 

.108 

.048 

Using  the  data  in  all  of  the  above  tables,  the  proposal 
evaluation  hierarchy,  complete  with  priority  weights,  is 
reconstructed  in  Figure  4.2.1.  This  hierarchy  is  not  all 
that  is  required  to  calculate  the  final  scores  for  each  of 
the  proposals.  With  procurement  regulations  prohibiting  the 
direct  comparison  of  different  contractor's  proposals,  an 
absolute  measurement  system  had  to  be  developed.  The  method 
adopted  in  this  study  discussed  in  Section  Three  was 
developed  by  Francis  and  Stonebraker  (Francis  and 
Stonebraker,  1991) .  Recall  that  with  this  method,  each 
proposal  is  rated  against  each  of  the  second  level 
subattributes,  then  each  proposal  is  assigned  the  priority 
weight  for  the  rating  it  receives.  The  final  score  is  then 
calculated  by  'folding  back  the  tree'  as  described  in 
Section  Three. 
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FIGURE  4,2.1. 
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The  tables  that  follow  give  the  results  of  this 
absolute  measurement  of  the  three  proposals.  Table  4.2.8 
gives  the  ratings  each  proposal  received  against  the  second 
level  subattributes,  and  Table  4.2.9  shows  how  the  final 
scores  will  map  into  an  overall  rating  for  the  proposals. 

TABLE  4.2.8. 


Proposal 

Rating 

FACTOR 

PROPOSAL 

EX 

AC 

MG 

UA 

1 

X 

Aircraft  Requirements 

2 

X 

3 

X 

1 

X 

Array  Configuration 

2 

X 

3 

X 

1 

X 

Digital  Control 

2 

X 

3 

X 

1 

X 

RF  Processing 

2 

X 

3 

X 

1 

X 

Miscellaneous 

2 

X 

3 

X 

FACTOR 


Analysis  Task 


Design  Task 


T  &  D  Task 


Analysis  Topics 


Design  Criteria 


T  &  D  Rec[uirements 


Technical  Discussion 
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TABLE  4.2.8.  Continued. 


TABLE  4.2.9 


Evaluations 

Resultcint 

Rating 

all  set  to: 

final  score 

Average 

Code  Range 

Exceptional 

.557 

.424 

.557  -  .424 

Acceptcible 

.291 

.199 

.199  -  .423 

Marginal 

.106 

.075 

.075  -  .198 

Unacceptable 

.043 

The  final  score  for  proposal  1  is  given  by: 


Proposal  1  =  .542C.344(.122)+.575(.297)+.526(.297)+.557(.183)+.527(.102)]  + 
.290[.595(.619)+.565(.284)+.575(.096)]  + 

.066t.584(.456)+.556(.235)+.513(.154)+.598(.114)+.510(.041)]  + 
.103[.547(.291)+.254(.122)+.507(.151)*.577(.128)+.521(.145)+.5B4(.163)] 
=  .543 


The  final  score  for  proposal  2  is  given  by: 


Proposal  2  =  .542[.485(.122)+.281(.297)+.103<.297)+.557(.183)+.325(.102)]  + 

.2901. 101 (. 619)+. 108<.284)+.120(. 096)}  + 

.066C.121(.456)+.306(.235)+.134(.154)+.598(.114)+.292(..;-  ))  + 
.103[.547(.291)+.578{.122)+.507(.151)+.577(.128)+.521(.1‘OV.  ..  f .  >63)] 
=  .269 

The  final  score  for  proposal  3  is  given  by: 


Proposal  3  =  .542[.485( .122)+.281( .297)+.324{ .297)+.293{ . 183)+.325( . 102)]  + 
.290[.101(.619)+.108(.284)+.120(.096)]  + 

.066C.121<.456)+.306<.235)+.134(.154)+.598<.114)+.292(.041))  + 
.103C.547(.291)+.578(.122)+.507(.151)+.577(.128)+.521{.145)+.584(.163)] 
=  .278 


Thus,  the  proposals  would  receive  the  following  ratings 


Proposal  1  =  Exceptional 


Proposal  2  =  Acceptable 


Proposal  3  =  Acceptable 
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V.  CONCLUSIONS 


5.I.  Conclusions 

This  study  examined  a  new  approach  to  resource 
allocation  and  proposal  evaluation  in  an  Air  Force  Research 
and  Development  (R  &  D)  organization.  A  method  for 
prioritizing  potential  new- start  programs  using  pairwise 
comparison  of  related  attributes  and  the  programs  was 
developed.  These  techniques,  along  with  an  absolute  rating 
system,  were  also  applied  to  technical  proposal  evaluations 
with  very  good  results.  In  both  cases,  the  rank  ordering  of 
projects  and  proposals  is  developed  in  a  highly  systematic 
fashion.  The  methodologies  developed  are  based  on  the 
Analytical  Hierarchy  Process  (AHP)  and  seem  to  offer  an 
effective  way  to  handle  the  resource  allocation  and  proposal 
evaluation  problems.  These  methods  were  designed  primarily 
for  use  in  R  &  D  organizations,  and  are  very  easily 
implemented.  Some  conclusions  as  a  result  of  this  study 
are: 


1.  The  AHP  can  be  easily  applied  to  resource 
allocation  and  proposal  evaluation  and  provides  an 
objective,  systematic,  and  easily  defensible  means  of 
prioritizing  projects  and  proposals. 
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2 .  The  present  guidelines  for  establishing  proposal 
evaluation  criteria  prescribed  in  current  procurement 
regulations  are  consistent  with  the  development  of  a 
hierarchical  model . 

3.  Translating  the  final  ntimerical  proposal  scores 
into  the  existing  ratings,  exceptional,  acceptable, 
marginal ,  and  unacceptable  makes  this  method  consistent 
with  established  procedures.  This  translation  can  also 
be  applied  to  the  level  one  attributes  Understanding 
the  Problem,  Soundness  of  Approach,  Compliance  with 
requirements,  and  Special  Technical  Factors,  if 
required. 

4.  Sensitivity  analysis  can  be  performed  on  the 
results  to  provide  greater  insight,  and  answer  all 
types  of  "what  if"  questions. 

5.  The  inclusion  of  an  absolute  measurement  system  in 
the  proposal  evaluation  stage  eliminates  the  need  to 
directly  compare  the  contractors  using  pairwise 
comparisons . 

5.2.  Future  Work 


To  be  consistent  with  present  proposal  evaluation 
procedures,  the  final  scores  obtained  for  the  proposals  were 
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mapped  into  the  present  ratings.  Work  needs  to  be  done  to 
develop  a  theoretically  sound  method  for  performing  this 
mapping . 

With  project  evaluation  in  the  resource  allocation 
stage,  the  managers  who  evaluate  the  projects  are  generally 
pressed  for  time;  therefore,  it  is  important  to  further 
simplify  the  instruction  booklet  and  the  project  evaluation 
questionnaire . 

The  attributes  used  in  the  resource  allocation 
hierarchy  are  the  same  as  those  used  in  the  present  process 
and  may  not  be  fully  independent.  It  would  be  interesting 
to  see  if  totally  independent  attributes  can  be  developed. 

Finally,  the  AHP  has  already  been  used  in  commercially 
available  software  packages;  however  these  are  not  geared 
toward  Air  Force  R  &  D.  Thus,  it  would  be  very  useful  if 
these  methods  could  be  incorporated  into  a  user  friendly 
computer  software  package  for  use  in  these  organizations. 
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EXPLANATION  OF  ATTRIBUTES  USED  IN  THIS  STUDY 


RESOURCE  ALLOCATION  EVALUATION  HIERARCHY; 


ADDITIONAL  -  Additional  factors  generally  accepted  as  being 
important  to  a  project.  The  following  subattributes  were 
considered  important  in  this  study:  follow-on  program, 
joint  program,  in-house,  supports  other  programs,  supports 
forecast  II. 

ALTERNATIVES  -  This  refers  to  the  fact  that  there  may  be 
more  than  one  alternative  to  solve  an  individual  need.  The 
following  subattributes  were  considered  important  in  this 
study:  none,  others  -  this  better,  several  equal,  many  - 

some  better. 

FEASIBILITY  -  A  measure  of  whether  the  project  will  solve 
the  given  problem  and  whether  management  will  be  receptive 
of  the  solution. 

FOLLOW-ON  -  This  refers  to  whether  or  not  the  proposed 
project  is  a  follow-on  from  a  successful  program. 

GENERALLY  APPLICABLE  -  CRITICAL  -  The  proposed  project  is 
generally  applicable  to  a  variety  of  critical  threats. 
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GENERALLY  APPLICABLE  -  NON-CRITICAL  -  The  proposed  project 
is  generally  applicable  to  a  variety  of  non- critical 
threats . 

HIGH  PAYOFF  -  The  proposed  project  offers  a  high  payoff 
approach  to  a  new  area. 

IN-HOUSE  -  If  resources  and  manpower  are  available,  the 
project  can  be  done  in-house. 

INNOVATION  -  This  refers  to  the  uniqueness  of  the  approach 
proposed  by  the  project  engineer.  The  following 
subattributes  were  considered  important  in  this  study:  high 
payoff  approach,  novel  approach,  some  new  ideas,  routine. 

JOINT  PROGRAM  -  The  proposed  project  will  be  done  in 
conjunction  with, or  with  assistance  from,  the  other  military 
services . 

LIMITED  CRITICAL  -  The  proposed  project  has  limited 
applicability  to  critical  threats. 

LIMITED  NON-CRITICAL  -  The  proposed  project  has  limited 
applicability  to  non- critical  threats. 
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MANY/SOME  -  Many  other  programs  were  proposed  and  some  were 
better. 

need  -  This  refers  to  the  stated  or  perceived  need  for  the 
proposed  project.  The  following  subattributes  were 
considered  important  in  this  study:  generally  critical, 
limited  critical,  generally  non- critical ,  limited  non- 
critical . 

NONE  -  This  is  the  only  program  proposed  to  meet  the  given 
need. 

NOVEL  APPROACH  -  The  proposed  project  offers  a  novel 
approach  to  an  old  area. 

OTHERS/THIS  BETTER  -  Similar  programs  were  proposed  to  meet 
the  given  need,  but  this  is  the  better  project. 

PRACTICALITY  -  Are  the  funds /manpower/ resources  available  to 
achieve  the  program  goal . 

PROJECT  DEFINITION  -  This  is  a  measure  of  how  well  the 
engineer  defines  the  problem  and  how  the  problem  will  be 


solved. 
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ROUTINE  -  This  refers  to  whether  the  proposed  project  is 
routine  and  offers  no  new  ideas. 

SEVERAL/EQUAL  -  Several  programs  were  proposed  and  they  are 
all  roughly  equal. 

SOME  NEW  IDEAS  -  This  refers  to  whether  the  proposed  project 
offers  some  new  ideas. 

SUPPORTS  FORECAST  II  -  The  proposed  project  supports 
FORECAST  II, 

SUPPORTS  OTHERS  -  The  proposed  effort  supports  existing  or 
future  programs. 

TECHNICAL  APPROACH  -  This  relates  to  the  approach  the 
project  engineer  proposes  to  meet  the  need/requirement .  The 
following  subattributes  were  considered  important  in  this 
study:  project  definition,  practicality,  feasibility. 


PROPOSAL  EVALUATION  HIERARCHY; 


ADVANCED  PROCESSING  INVESTIGATION  -  The  contractor  must  have 
corporate  experience  and  individual  experience  in  advanced 
real  tine  processing. 
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AIRCRAFT  REQUIREMENTS  -  Includes  aircraft,  aircraft 
subsystem,  and  environment  requirements. 

ANALYSIS  TASK  -  the  plan  for  performing  analysis,  extracting 
important  data,  performing  trade-offs,  evaluating  data,  and 
interpreting  results. 

ANALYSIS  TOPICS  -  The  proposed  effort  must  address  minimum 
analysis  topics  unless  otherwise  justified  in  the  proposal. 

ARRAY  CONFIGURATION  TRADE-OFFS  -  Includes  affects  aperture 
technologies  have  on  placement  and  design  of  the  antenna 
array . 

COMPLIANCE  WITH  REQUIREMENTS  -  The  contractor  shall  comply 
with  the  requirements  of  the  Statement  of  Work  and  the 
Request  for  Proposal.  The  following  subattributes  were 
considered  important  in  this  study:  analysis  topics,  design 
criteria,  testing  and  demonstration  requirements,  technical 
discussion,  and  presentation. 

DESIGN  CRITERIA  -  The  proposed  effort  will  meet  minimum 
design  criteria  unless  otherwise  justified  in  the  proposal. 
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DESIGN  TASK  -  The  contractor  must  show  a  plan  for  creating  a 
top-level  design  of  a  control  system  and  associated  RF 
switching  and  array  configuration. 

DIGITAL  CONTROL  -  Short  for  processing  and  digital  control 
of  high  speed,  real-time  RF  applications. 

DIGITAL  CONTROLLER  DEVELOPMENT  -  The  contractor  must  show 
corporate  and  individual  experience  in  digital  control  and 
control  system  development. 

ECM  SYSTEM  DEVELOPMENT  -  The  contractor  must  have  individual 
and  corporate  experience  in  ECM  systems. 

EMBEDDED  SYSTEM  SOFTWARE  AND  ALGORITHM  (ESS  &  A) 

DEVELOPMENT  -  Must  haze  corporate  and  individual  experience 
in  signal  processing  algorithm  development  and  software 
development . 

MISCELLANEOUS  -  Miscellaneous  device  technology,  power  and 
heat  distribution  issues.  Also  includes  logistics  and 
manufacturing  technologies . 

MMIC  APPLICATION  -  The  contractor  must  have  individual  and 
corporate  experience  in  MMIC  device  application. 
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PRESENTATION  -  Proposals  must  be  neat,  organized,  readable, 
understandable,  and  grammatically  correct. 

RF  PROCESSING  -  Includes  knowledge  of  current  technologies 
and  general  RF  system  design. 

SOLID  STATE  (SS)  ARRAY  DESIGN  -  The  contractor  must  have 
corporate  and  individual  experience  technology  and  design  of 
solid  state  arrays. 


SOUNDNESS  OF  APPROACH  -  The  contractors  will  be  evaluated  on 
the  soundness  of  their  approach  towards  providing  a  solution 
to  the  problem.  The  following  subattributes  were  considered 
important  in  this  study;  analysis  task,  design  task,  and 
testing  and  demonstration  task. 

SPECIAL  TECHNICAL  FACTORS  -  The  contractor  will  be  evaluated 
on  experience  and  individual  expertise  in  related 
technologies.  The  following  subattributes  were  considered 
important  in  this  study:  solid  state  array  design,  ECM 
system  development,  digital  controller  development,  advanced 
processing  investigation,  embedded  system  software  and 
algorithm  development,  and  P®!IC  device  application. 

TECHNICAL  DISCUSSION  -  A  measure  to  ensure  the  proposal  does 
not  restate  the  Statement  of  Work,  but  adds  technical 
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discussion  pertinent  to  the  understanding  and  successful 
solution  of  the  problem. 

TEST  AND  DEMONSTRATION  REQUIREMENT  -  The  proposed  effort 
must  meet  minimum  test  and  demonstration  requirements  unless 
otherwise  justified  in  the  proposal. 

TEST  AND  DEMONSTRATION  TASK  -  The  proposed  effort 
must  show  a  low  risk  plan  for  testing  critical  features  of 
the  design.  Considered  a  risk  reduction  task,  hence  the 
test  plan  should  reflect  identifying  risky  areas  of  design 
for  test  and  evaluation. 

UNDERSTANDING  THE  PROBLEM  -  The  contractor  must  show  in 
their  proposal  an  understanding  of  the  problem  as  related  to 
the  elements  under  this  factor. 
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Appendix  B 

Introduction  to  Questionnaire 


86 


A  PROCEDURE  FOR  EVALUATING  TECHNICAL  PROPOSALS 


INSTRUCTIONS  BOOKLET 


By 

Robert  J.  Keppler 
February  1993 


UNIVERSITY  OF  WISCONSIN  AT  MILWAUKEE 
DEPARTMENT  OF  INDUSTRIAL  AND  SYSTEMS  ENGINEERING 


87 


OBJECTIVE  AND  SCOPE  OF  RESEARCH i 


The  purpose  of  this  study  is  to  determine  the  most 
appropriate  attributes  that  are  associated  with  the 
evaluation  of  technical  proposals  and  to  formulate  a 
procedure  to  evaluate  and  prioritize  engineering  proposals 
offered  by  contractors. 

While  the  methods  for  evaluated  proposals  are  well 
established  and  documented,  the  rating  a  proposal  receives 
is  sometimes  arbitrarily  or  subjectively  determined.  The 
method  proposed  in  this  study  will  yield  a  systematically 
determined  rating  for  each  proposal.  Also,  the 
determination  and  ranking  of  the  attributes  used  in  the 
proposal  evaluation  can  be  done  early  in  the  procurement 
process  and  included  in  the  Request  for  Proposal  package 
sent  to  prospective  offerors.  This  will  allow  the 
contractors  to  know  which  attributes  are  and  their  ranking 
in  relation  to  one  another. 

For  the  purpose  of  evaluating  proposals  a  multi - 
attribute  evaluation  model  based  on  the  analytical  hierarchy 
process  is  developed.  The  analytical  hierarchy  process  was 
developed  by  Thomas  Saaty  in  1972  and  has  been  applied  to  a 
wide  variety  of  problems,  including  capital  medical 
equipment  replacement  in  hospitals  (Lo,  1992)  and 
determining  (after  the  fact)  the  best  course  of  action  to 
follow  in  the  Iranian  Hostage  Crisis  (Saaty,  et  al,  1985). 


This  model  considers  twenty  three  attributes  based  on 
evaluation  criteria  I  obtained  Kevin  Geiger,  WL/AAWW.  In 
order  to  demonstrate  application  of  the  model,  this  study 
will  examine  three  proposals  that  were  sent  in  for  the 
Antenna  Element  Manager  program.  The  model  follows  a  two 
step  process:  first,  data  on  preferences  among  the  various 
attributes  is  gathered  through  the  use  of  pairwise 
comparisons  and  a  questionnaire;  second,  the  pairwise 
comparison  data  is  used  to  determine  the  relative  weights  of 
all  the  attributes,  the  weights  are  then  used  to  rank  the 
proposals . 
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Questionnaire  Instructions 

These  instructions  will  explain  how  the  tables  included  in 
this  questionnaire  need  to  be  filled  out.  Please  read  them 
carefully  before  attempting  to  fill  out  the  tables. 
Procedure: 

Please  indicate  how  the  following  factors  listed  under 
A  and  B  rank  in  importance  to  one  another  by  placing  a  check 
mark  under  the  rating  on  the  appropriate  half  of  the  table. 
Thus  when  comparing  factor  UUDERSTANDING  THE  PROBLEM  under 
column  A  with  factor  SOUNDNESS  OF  APPROACH  under  column  B, 
if  UNDERSTANDING  THE  PROBLEM  is  more  important  as  compared 
to  SOUNDNESS  OF  APPROACH  put  a  check  mark  under  the 
appropriate  number  in  the  left  half  of  the  table.  See  the 
scale  provided  below  to  help  determine  how  much  more 
important  UNDERSTANDING  THE  PROBLEM  is  in  comparison  with 
SOUNDNESS  OF  APPROACH. 

Similarly,  if  you  determine  SOUNDNESS  OF  APPROACH  is 
more  important  than  UNDERSTANDING  THE  PROBLEM,  put  the  check 
mark  under  the  appropriate  number  on  the  right  half  of  the 
table.  Again,  how  much  more  important  SOUNDNESS  OF  APPROACH 
is  as  compared  to  UNDERSTANDING  THE  PROBLEM  is  determined  by 
selecting  the  proper  number  from  the  scale  provided  below. 
Please  remember  that  there  can  be  only  one  check  mark  (on 
the  appropriate  side  of  the  table)  along  one  row  when 
comparing  the  two  attributes.  Continue  to  fill  out  each  row 


in  a  given  table  in  a  similar  manner. 
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SCALE  FOR  COMPARISCr 


Intensity  of 
Importcince 

Definition 

Explanation 

1 

Equal  importance  of  both 
elements 

Two  elements  contribute 
equally  to  the  property 

3 

Weak  importance  of  one 
element  over  cuiother 

Experience  and  judgement 
slightly  favor  one  over 
another 

5 

Essential  or  strong 
importance  of  one  element 

Experience  and  judgement 
strongly  favor  one  element 
over  amother 

7 

Demonstrated  importance 
of  one  element  over 
another 

An  element  is  strongly 
favored  and  its  dominance 
demonstrated  in  practice 

9 

Absolute  importsmce  of 
one  element  over  another 

The  evidence  favoring  one 
element  over  another  is  of 
the  highest  possible  order 
of  affirmation 

2,4, 6, 8 

Intermediate  values 
between  two  adjacent 
judgements 

Compromise  is  needed 
between  two  judgements 

The  following  example  will  illustrate  the  above  comparison 
methodology. 

Exeunple ; 

Assume  you  are  provided  the  following  table  and  asked 
to  make  a  comparison  between  the  two  factors  UNDERSTANDING 
THE  PROBLEM  and  SOUNDNESS  OF  APPROACH. 

Step  1 

Determine  if  UNDERSTANDING  THE  PROBLEM  is  more 
important  than  SOUNDNESS  OF  APPROACH,  or  vice  versa. 

Step  2 


For  the  sake  of  this  example,  it  is  assumed  that 
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UNDERSTANDING  THE  PROBLEM  is  more  important  than  SOUNDNESS 
OF  APPROACH,  thus  the  left  half  of  the  table  will  be  used. 
Step  3 

Choose  a  number  from  the  above  scale  which  best 
represents  your  judgement  in  terms  of  the  importance  of 
UNDERSTANDING  THE  PROBLEM  over  SOUNDNESS  OF  APPROACH.  Place 
a  check  mark  under  the  number  in  the  appropriate  half  of  the 
table.  Assume  the  number  is  8.  Because  it  has  already  been 
determined  that  UNDERSTANDING  THE  PROBLEM  is  more  important 
than  SOUNDNESS  OF  APPROACH,  place  a  check  mark  under  8  on 
the  left  half  of  the  table.  Proceed  to  carefully  put  check 
marks  in  all  rows  in  a  table  until  you  reach  the  end. 


A 

3 

B 

7 

6 

5 

i 

3 

2 

1 

1 

2 

3 

<k 

5 

6 

7 

B 

9 

B 

UNDERSTANDING 

1 

I 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

SOUNDNESS 

If  SOUNDNESS  OF  APPROACH  is  the  more  important  factor,  we 
have  the  following  table: 


A 

9 

8 

7 

6 

5 

3 

3 

2 

L 

2 

L 

L 

6 

L 

B 

L 

B 

UNDERSTANDING 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

SOUNDNESS 

Note:  A  full  explanation  of  the  attributes  found  in  the  following 


tables  is  listed  at  the  end  of  the  questionnaire. 
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Appendix  C 

Questionnaire  and  Cco^jarison  Table  for 
Resource  Allocation 


PRELIMINARY  QUESTIONNAIRE 
AND 

COMPARISONS  TABLE 
FOR 

EVALUATION  OF  PROJECTS  BEFORE  THE  ANALYSIS  RAC 


Piojse  return  this  booklet  after  filling  the  questionnaire  and  tables. 
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PRELIMINOR/  INPCFM\TICM 


Narc  Title 

Office  Svnrtx)! 

^dress 

Phone  Date 


Note: 

Please  cc^lete  this  data  sheet  and  return  with  corpleted  table.  This 
infomnation  wi  1 1  be  kept  ccnrpletely  confidential.  Feel  free  to  offer 
any  comrents  about  this  model  you  dean  appropriate.  Thank  you  very  rruch 
for  your  time! 
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TABLE  FOR  FACTORS  ONDBR  PROJECT  BVAUIATION: 


Please  indicate  how  the  following  factors  listed  under  A  cuid  B  rcink  in 
importance  to  one  another  by  placing  a  check  mark  \inder  the  rating  on 
the  appropriate  half  of  the  tcJale. 


A 

9 

8 

7 

6 

5 

i 

3 

2 

L 

1 

2 

3 

i 

S 

s 

7 

8 

9 

B 

NEED 

TECH.  APPROACH 

NEED 

ADDITIONAL 

NEED 

ALTERNATIVES 

NEED 

INNOVATION 

TECH.  APPROACH 

ADDITIONAL 

TECH.  APPROACH 

ALTERNATIVES 

TECH.  APPROACH 

INNOVATION 

ADDITIONAL 

ALTERNATIVES 

ADDITIONAL 

INNOVATION 

ALTERNATIVES 

INNOVATION 

Table  1:  Factors  Under  Project  Evaluation 


EXPLANATION  OF  FACTORS  -  TABLE  1 

NEED  -  This  refers  to  the  stated  or  perceived  need  for  the 
proposed  project. 

TECH.  APPROACH  -  Short  for  TECHNICAL  APPROACH.  This  relates 
to  the  approach  the  project  engineer  proposes  to  meet  the 
need/ requirement . 

ADDITIONAL  -  Additional  factors  generally  accepted  as  being 
important  to  a  project. 

ALTERNATIVES  -  This  refers  to  the  fact  that  there  may  be 
more  than  one  alternative  to  solve  an  individual  need. 

INNOVATION  -  This  refers  to  the  uniqueness  of  the  approach 
proposed  by  the  project  engineer. 


97 


TABLE  FOR  FACTORS  UNDER  TECHNICAL  APPROACH; 

Please  indicate  how  the  following  factors  listed  under  A  and 
B  rank  in  importance  to  one  another  by  placing  a  check  mark 
under  the  rating  on  the  appropriate  half  of  the  table. 


A 

9 

3 

7 

S 

5 

3 

2 

L 

1 

2 

3 

i 

5 

S 

7 

3 

3 

B 

PROJECT  DEFINITION 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

PRACTICALITY 

PROJECT  DEFINITION 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

FEASIBILITY 

PRACTICALITY 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

FEASIBILITY 

Table  2;  Factors  Under  Tecdinical  Approach 
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EXPLANATION  OF  FACTORS  -  TABLE  2 

PROJECT  DEFINITION  -  This  is  a  measure  of  how  well  the 
engineer  defines  the  problem  and  how  the  problem  will  be 
solved. 

PRACTICALITY  -  Are  the  funds/manpower/resources  available  to 
achieve  the  program  goal . 

FEASIBILITY  -  A  measure  of  whether  the  project  will  solve 
the  given  problem  and  whether  management  will  be  receptive 
of  the  solution. 
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TABLE  FOR  FACTORS  UNDER  NEED; 

Please  indicate  how  the  following  factors  listed  under  A  and 
B  rank  in  importance  to  one  another  by  placing  a  check  mark 
under  the  rating  on  the  appropriate  half  of  the  table. 


GEN  CRITICAL 

LIMITED  CRITI 

CAL 

GEN  CRITICAL 

GEN  NON- CRITI 

15 

GEN  CRITICAL 

IIIIIIIIIIIIIIIIIIBaBS 

LIMITED  CRITICAL 

llllllllllllllllll 

GEN  NON -CRITI 

LIMITED  CRITICAL 

imimmiiiiiiiPiMii! 

GEN  NON- CRITICAL 

iiiiiiiiimiimiiiiMii 
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EXPLANATION  OF  FACTORS  -  TABLE  3 

GEN  CRITICAL  -  Abbreviation  for  GENERALLY  APPLICABLE  - 
CRITICAL.  The  proposed  project  is  generally  applicable  to  a 
variety  of  critical  threats. 

LIMITED  CRITICAL  -  Short  for  LIMITED  APPLICABILITY  - 
CRITICAL.  The  proposed  project  has  limited  applicability  to 
critical  threats. 

GEN  NON- CRITICAL  -  Abbreviation  for  GENERALLY  APPLICABLE  - 
NON-CRITICAL.  The  proposed  project  is  generally  applicable 
to  a  variety  of  non-critical  threats. 

LIMITED  NON-CRIT  -  Abbreviation  of  LIMITED  APPLICABILITY  - 
NON-CRITICAL.  The  proposed  project  has  limited 
applicability  to  non-critical  threats. 
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TABLE  FOR  FACTORS  UNDER  INNOVATION; 

Please  indicate  how  the  following  factors  listed  under  A  and 
B  rank  in  importance  to  one  another  by  placing  a  check  mark 
under  the  rating  on  the  appropriate  half  of  the  table. 


A 

9 

8 

7 

6 

5 

i 

3 

2 

L 

1 

2 

3 

i 

5 

6 

7 

3 

9 

B 

HIGH  PAYOFF 

NOVEL  APPROACH 

HIGH  PAYOFF 

SOME  NEW  IDEAS 

HIGH  PAYOFF 

ROUTINE 

NOVEL  APPROACH 

SOME  NEW  IDEAS 

NOVEL  APPROACH 

ROUTINE 

SOME  NKr  IDEAS 

ROUTINE 

Table  4:  Factors  Iftider  Innovation 
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EXPLANATION  OF  FACTORS  -  TABLE  4 

HIGH  PAYOFF  -  The  proposed  project  offers  a  high  payoff 
approach  to  a  new  area. 

NOVEL  APPROACH  -  The  proposed  project  offers  a  novel 
approach  to  an  old  area. 

SOME  NEW  IDEAS  -  This  refers  to  whether  the  proposed  project 
offers  some  new  ideas. 

ROUTINE  -  This  refers  to  whether  the  proposed  project  is 
routine  and  offers  no  new  ideas. 
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TABLE  FOR  FACTORS  UNDER  ALTERNATIVES t 

Please  indicate  how  the  following  factors  listed  under  A  and 
B  rank  in  importance  to  one  another  by  placing  a  check  mark 
under  the  rating  on  the  appropriate  half  of  the  table. 


B 

NONE 

NONE 

iiiimiiiiiiiiiii 

SEVERAL/EQUAL 

NONE 

imiiiiiiiiiiiiii 

MANY /SOME  BETTER 

OTHERS /BETTER 

SEVERAL/EQUAL 

OTHERS /BETTER 

iiiiiiiiiiiiiiiiii 

MANY/SOME  BETTER 

SEVERAL/EQUAIi 

IIIIIIIIIIIIIIIIII 

MANY /SOME  BETTER 

Table  5:  Factors  Under  Alternatives 
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EXPLANATION  OF  FACTORS  -  TABLE  5 


NONE  -  This  is  the  only  program  proposed  to  meet  the  given 
need. 

OTHERS/THIS  BETTER  -  Similar  programs  were  proposed  to  meet 
the  given  need,  but  this  is  the  better  project. 

SEVERAL/EQUAL  -  Several  programs  were  proposed  and  they  are 
all  roughly  equal. 

many/some  -  Many  other  programs  were  proposed  and  some  were 


better. 
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TABLE  FOR  FACTORS  UNDER  ADDITIONAL; 

Please  indicate  how  the  following  factors  listed  under  A  and 
B  rank  in  importance  to  one  another  by  placing  a  check  mark 
under  the  rating  on  the  appropriate  half  of  the  table. 


A 

9 

8 

7 

S 

5 

i 

3 

2 

1 

1 

2 

3 

i 

5 

S 

7 

S 

9 

B 

FOLLOW-ON 

JOINT  PROGRAM 

FOLLOW-ON 

IN-HOUSE 

FOLLOW-ON 

SUPPORTS  OTHERS 

FOLLOW-ON 

JOINT  PROGRAM 

IN-HOUSE 

JOINT  PROGRAM 

SUPPORTS  OTHERS 

JOINT  PROGRAM 

— 

IN-HOUSE 

SUPPORTS  OTHERS 

IN-HOUSE 

SUPPORTS  OTHERS 

Table  6:  Factors  Under  Additional 
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EXPLANATION  OF  FACTORS  -  TABLE  6 

FOLLOW-ON  -  This  refers  to  whether  or  not  the  proposed 
project  is  a  follow-on  from  a  successful  program. 

JOINT  PROGRAM  -  The  proposed  project  will  be  done  in 
conjunction  with,  or  with  assistance  from,  the  other 
military  services. 

IN-HOUSE  -  If  resources  and  manpower  are  available,  the 
project  can  be  done  in-house. 

SUPPORTS  OTHERS  -  The  proposed  effort  supports  existing  or 
future  programs . 

SUPRTS  FORECAST  -  Short  for  SUPPORTS  FORECAST  II.  The 
proposed  project  supports  FORECAST  II. 
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PAIRWISE  COMPARISON  OF  TECHNICAL  PROJECTS 


The  pairwise  comparisons  in  the  tables  that  follow  are 
to  be  used  to  evaluat:e  the  projects  based  on  the  previously 
ranked  attributes.  As  with  the  comparisons  just  completed, 
these  will  be  used  to  develop  the  weights  that  will  be  used 
in  ranking  the  projects. 
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E 


I.  The  following  comparisons  are  related  to  NEED. 

1.  From  the  view  point  of  GENERALLY  APPLICABLE  -  CRITICAL  please 
indicate  the  importance  of  one  project  vs  the  others  in  the  table. 


B 

PROJECT 

2 

PROJECT 

3 

PROJECT 

3 

2.  From  the  view  point  of  LIMITED  APPLICABILITY  -  CRITICAL  please 
indicate  the  importance  of  one  project  vs  the  others  in  the  table. 


PROJECT  1 


PROJECT  1 


PROJECT  2 


3.  From  the  view  point  of  GENERALLY  APPLICABLE  -  NON  CRITICAL  please 
indicate  the  importance  of  one  project  vs  the  others  in  the  table. 


B 

PROJECT 

2 

PROJECT 

3 

PROJECT 

3 

A 

PROJECT 

1 

PROJECT 

1 

PROJECT 

2 

B 

PROJECT 

2 

PROJECT 

3 

PROJECT 

3 

4 .  From  the  view  point  of  LIMITED  APPLICABILITY  -  NON  CRITICAL  please 
indicate  the  importance  of  one  project  vs  the  others  in  the  table. 


PROJECT 

1 

PROJECT 

1 

PROJECT 

2 

B 

PROJECT 

2 

PROJECT 

3 

PROJECT 

3 
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II.  The  following  comparisons  are  related  to  TECHNICAL  APPROACH. 
1 .  From  the  view  point  of  PROJECT  DEFINITION  please  indicate  the 
importance  of  one  project  vs  the  others  in  the  tcible. 


A 

8 

7 

S 

5 

i 

3 

2 

1 

1 

2 

3 

i 

5 

g 

7 

a 

3 

B 

PROJECT 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

PROJECT 

2 

PROJECT 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

PROJECT 

3 

PROJECT 

2 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

PROJECT 

3 

2.  From  the  view  point  of  PRACTICALITY  please  indicate  the  importance 
of  one  project  vs  the  others  in  the  table. 


A 

3 

8 

7 

g 

5 

3 

2 

1 

1 

2 

3 

i 

5 

g 

7 

8 

3 

B 

PROJECT 

I 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

PROJECT 

2 

PROJECT 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

PROJECT 

3 

PROJECT 

2 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

PROJECT 

3 

3 .  From  the  view  point  of  FEASIBILITY  please  indicate  the  importance  of 
one  project  vs  the  others  in  the  table. 
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III.  The  following  comparisons  are  related  to  INNOVATION . 

1.  From  the  view  point  of  HIGH  PAYOFF  APPROACH  TO  NEW  AREA  please 
indicate  the  importance  of  one  project  vs  the  others  in  the  table. 


2 .  From  the  view  point  of  NOVEL  APPROACH  TO  OLD  AREA  please  indicate 
the  importcince  of  one  project  vs  the  others  in  the  table. 


3 .  From  the  view  point  of  SOME  NEW  IDEAS  please  indicate  the  importance 
of  one  project  vs  the  others  in  the  table. 


Ill 


IV.  The  following  comparisons  are  related  to  ALTERNATIVES . 

1 .  From  the  view  point  of  NONE  please  indicate  the  importance  of  one 
project  vs  the  others  in  the  tcible. 


2 .  From  the  view  point  of  OTHERS  -  BUT  THIS  IS  BETTER  please  indicate 
the  importance  of  one  project  vs  the  others  in  the  tadjle. 


3 .  From  the  view  point  of  SEVERAL  OF  EQUAL  VALUE  please  indicate  the 
importance  of  one  project  vs  the  others  in  the  table. 

B 

PROJECT  2 
PROJECT  3 
PROJECT  3 


4 .  From  the  view  point  of  MANY  OTHERS  -  SOME  BETTER  please  indicate  the 
importance  of  one  project  vs  the  others  in  the  table. 


112 


V.  The  following  comparisons  are  related  to  ADDITIOMAL . 

1.  From  the  view  point  of  FOLLOW-ON  FROM  SUCCESSFUL  PROGRAM  please 
indicate  the  importance  of  one  project  vs  the  others  in  the  table. 


PROJECT  1 


PROJECT  1 


PROJECT  2 


2 .  From  the  view  point  of  JOINT  PROGRAM  please  indicate  the  importance 
of  one  project  vs  the  others  in  the  table. 


PROJECT  1 


PROJECT  1 


PROJECT  2 


PROJECT  2 


PROJECT  3 


PROJECT  3 


3 .  From  the  view  point  of  CAN  BE  DONE  IN-HOUSE  please  indicate  the 
importance  of  one  project  vs  the  others  in  the  tcible. 


A 

PROJECT 

1 

PROJECT 

1 

PROJECT 

2 

4 .  From  the  view  point  of  SUPPORTS  EXISTING  OR  FUTURE  PROGRAMS  please 
indicate  the  importance  of  one  project  vs  the  others  in  the  table. 


PROJECT  1 


PROJECT  1 


PROJECT  2 


B 

PROJECT 

2 

PROJECT 

3 

PROJECT 

3 

5.  From  the  view  point  of  SUPPORTS  FORECAST  II  please  indicate  the 
importcuice  of  one  project  vs  the  others  in  the  table. 
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Appendix  D 

Questionnaire  and  Comparison  Table  for 
Proposal  Evaluation 


PRELIMINARY  QUESTIONNAIRE 
AND 

COMPARISONS  TABLE 
FOR 

EVALUATION  OF  TECHNICAL  PROPOSALS 


Please  return  this  booklet  after  filling  the  questionnaire  and  tables 
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PRBLIMIMARY  INFORMATION 

Name  Title 

Office  Symbol 

Address 

Phone  Date 


Note : 

Please  complete  this  data  sheet  and  return  with  con^leted  table.  This 
information  will  be  kept  completely  confidential.  Feel  free  to  offer 
any  comments  about  this  model  you  deem  appropriate .  Thank  you  very  much 
for  your  time! 
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TABLE  FOR  FACTORS  PMDBR  PROJECT  BVMJ1A.TION: 

Please  indicate  how  the  following  factors  listed  under  A  and  B  rank  in 
importance  to  one  another  by  placing  a  check  mark  under  the  rating  on 
the  appropriate  half  of  the  table. 


A 

9 

B 

7 

S 

5 

i 

3 

2 

1 

2 

3 

i 

5 

S 

7 

8 

9 

B 

UNDERSTANDING 

SOUNDNESS 

UNDERSTANDING 

COMPLIANCE 

UNDERSTANDING 

SPECIAL  FACTORS 

SOUNDNESS 

COMPLIANCE 

SOUNDNESS 

SPECIAL  FACTORS 

COMPLIANCE 

SPECITUi  FACTORS 

Table  1:  Factors  Dnder  Project  Evaluation 
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EXPLANATION  OF  FACTORS  -  TABLE  1 


UNDERSTANDING  -  Short  for  UNDERSTANDING  THE  PROBLEM.  The 
contractor  must  show  in  their  proposal  an  understanding  of 
the  problem  as  related  to  the  elements  under  this  factor. 

SOUNDNESS  -  Short  for  SOUNDNESS  OF  APPROACH.  The 
contractors  will  be  evaluated  on  the  soundness  of  their 
approach  towards  providing  a  solution  to  the  problem, 

COMPLIANCE  -  Short  for  COMPLIANCE  WITH  REQUIREMENTS.  The 
contractor  shall  comply  with  the  requirements  of  the 
Statement  of  Work  or  Request  for  Proposal. 

SPECIAL  FACTORS  -  Short  for  SPECIAL  TECHNICAL  FACTORS.  The 
contractor  will  be  evaluated  on  experience  and  individual 
expertise  in  related  technologies. 
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TABLE  FOR  FACTORS  UNDER  UNDERSTANDING  THE  PROBLEM; 

Please  indicate  how  the  following  factors  listed  under  A  and 
B  rank  in  importance  to  one  another  by  placing  a  check  mark 
under  the  rating  on  the  appropriate  half  of  the  table. 


AIRCRAFT  RQMTS 

ARRAY  CONFIG 

AIRCRAFT  RQMTS 

DIGITAL  CONTROL 

AIRCRAFT  RQMTS 

RF  PROCESSING 

AIRCRAFT  RQMTS 

MISCELLANEOUS 

ARRAY  CONFIG 

DIGITAL  CONTROL 

ARRAY  CONFIG 

RF  PROCESSING 

ARRAY  CONFIG 

MISCELLANEOUS 

DIGITAL  CONTROL 

RF  PROCESSING 

DIGITAL  CONTROL 

MISCELLANEOUS 

RF  PROCESSING 

MISCELLANEOUS 

Table  2:  Factors  Under  Understanding  the  Problem 
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EXPLANATION  OF  FACTORS  -  TABLE  2 


AIRCRAFT  RQHTS  -  Includes  aircraft,  aircraft  subsystem  and 
environmental  requirements. 


ARRAY  CONFIG  -  Short  for  ARRAY  ANTENNA  CONFIGURATION  TRADE¬ 
OFFS.  Includes  affects  aperture  technologies  have  in  the 
placement  and  design  of  the  array. 


DIGITAL  CONTROL  -  Short  for  PROCESSING  AND  DIGITAL  CONTROL 
OF  HIGH  SPEED,  REAL-TIME,  RF  APPLICATIONS. 

RF  PROCESSING  -  Short  for  RF  SIGNAL  PROCESSING  AND 
SWITCHING.  Includes  knowledge  of  current  technologies  and 
general  RF  system  design. 


MISCELLANEOUS  -  Short  for  MISCELLANEOUS  DEVICE  TECHNOLOGY, 
POWER  AND  HEAT  DISTRIBUTION  ISSUES.  Includes  logistics  and 
manufacturing  technologies. 
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TABLE  FOR  FACTORS  UNDER  SOTTNDNESS  OF  APPROACH; 

Please  indicate  how  the  following  factors  listed  under  A  and 
B  rank  in  importance  to  one  another  by  placing  a  check  mark 
under  the  rating  on  the  appropriate  half  of  the  table. 


A 

3 

8 

7 

6 

S 

1 

3 

2 

1 

1 

2 

3 

i 

5 

5 

7 

8 

3 

B 

ANALYSIS  TASK 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

DESIGN  TASK 

ANALYSIS  TASK 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

T  &  D  TASK 

DESIGN  TASK 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

T  &  D  TASK 

Table  3 :  Factors  Itader  Soundness  of  T^jproach 
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EXPLANATION  OF  FACTORS  -  TABLE  3 

ANALYSIS  TASK  -  Must  show  a  plan  for  performing  analysis, 
extracting  important  data,  performing  trade-offs,  evaluating 
data,  and  interpreting  results. 

DESIGN  TASK  -  Must  show  a  plan  for  creating  a  top  level 
design  of  control  system  and  associated  RF  switching  and 
array  configuration. 

T  &  D  TASK  -  Short  for  TESTING  AND  DEMONSTRATION  TASK.  Must 
show  a  low  risk  plan  for  testing  critical  features  of  the 
design.  Considered  a  risk  reduction  task,  hence  the  test 
plan  should  reflect  identifying  risky  areas  of  design  for 


test  and  evaluation. 
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TABLE  FOR  FACTORS  UNDER  COMPLIANCE  WITH  REQUIREMENTS; 

Please  indicate  how  the  following  factors  listed  under  A  and 
B  rank  in  importance  to  one  another  by  placing  a  check  mark 
under  the  rating  on  the  appropriate  half  of  the  table. 


ANALYSIS  TOPICS 

IIIIIIIIIIIHIIIII 

DESIGN  CRITERIA 

ANALYSIS  TOPICS 

TEST  &  DEMO  RQMT 

ANALYSIS  TOPICS 

TECHNICAL  DISCUS 

ANALYSIS  TOPICS 

PRESENTATION 

DESIGN  CRITERl. 


TEST  &  DEMO  RQMT 


TECHNICAL  DISCUS 


PRESENTATION 


TECHNICAL  DISCUS 


PRESENTATION 


PRESENTATION 


Table  4 :  Factors  Under  Coaqpliance  with  Requirements 
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EXPLANATION  OF  FACTORS  -  TABLE  4 


ANALYSIS  TOPICS  -  The  proposed  effort  must  address  minimum 
analysis  topics  unless  otherwise  justified  in  the  proposal. 

DESIGN  CRITERIA  -  The  proposed  effort  will  meet  minimum 
design  criteria  unless  otherwise  justified  in  the  proposal. 

TEST  &  DEMO  RQHT  -  The  proposed  effort  must  meet  minimum 
test  and  demonstration  requirements  unless  otherwise 
specified  in  the  proposal. 

TECHNICAL  DISCUS  -  A  measure  to  ensure  the  proposal  does  not 
restate  the  Statement  of  Work  but  adds  technical  discussion 
pertinent  to  the  understanding  and  successful  solution  of 
the  problem. 


PRESENTATION  -  Proposals  must  be  neat,  organized,  readable, 
understandable,  and  grammatically  correct. 
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Table  for  Factors  Under  Special  Technical  Factors t 

Please  indicate  how  the  following  factors  listed  under  A  and 
B  rank  in  importance  to  one  another  by  placing  a  check  mark 
under  the  rating  on  the  appropriate  half  of  the  table. 


Table  5:  Factors  Under  Special  Technical  Factors 
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EXPLANATION  OP  FACTORS  -  TABLE  5 

SS  ARSAY  DESIGN  -  Abbreviation  for  SOLID-STATE  ARRAY  DESIGN. 
Rating  for  overall  and  individual  experience. 

ECM  SYSTEM  DEV  -  Must  have  corporate  and  individual 
experience  in  ECM  systems. 

DIGITAL  CONTROL  -  Short  for  DIGITAL  CONTROLLER  DEVELOPMENT. 
Must  have  corporate  and  individual  experience  in  digital 
control  and  control  system  development. 

ADVANCED  PROCESS  -  Short  for  ADVANCED  PROCESSING 
INVESTIGATION.  Must  have  corporate  and  individual 
experience  in  advanced  real  time  processing. 

ESS  &  A  DEV  -  Short  for  EMBEDDED  SYSTEM  SOFTWARE  AND 
ALGORITHM  DEVELOPMENT.  Must  have  corporate  and  individual 
experience  in  signal  processing  algorithm  development  and 
software  development. 

MMIC  APPLICATION  -  Must  have  corporate  and  individual 
experience  in  MMIC  device  application. 
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PAIRWISE  COMPARISON  OF  TECHNICAL  PROPOSAL  RATINGS 


The  pairwise  comparisons  in  the  tables  that  follow  are  to  be 
used  to  evaluate  the  ratings  that  are  typically  applied  to 
technical  proposals.  This  will  permit  item  specific 
meanings  for  the  proposal  ratings.  For  example,  the 
relative  importance  of  "exceptional"  and  "unacceptable" 
likely  will  be  different  when  rating  AIRCRAFT  PLATFORM 
REQUIREMENTS  versus  PRESENTATION. 
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I.  The  Following  Comparisons  are  related  to  UNDERSTANDING  THE  PROBLEM. 
1.  From  the  viewpoint  of  AIRCRAFT  REQUIREMENTS,  please  indicate  the 
importance  of  one  rating  over  the  others  in  the  table. 


2 .  From  the  view  point  of  ARRAY  CONFIGURATION, 
importcince  of  one  rating  vs  the  others  in  the  table 


EXCEPTIONAL 


EXCEPTIONAL 


EXCEPTIONAL 


ACCEPTABLE 


ACCEPTABLE 


MARGINAL 


please  indicate  the 


B 

II 

ACCEPTABLE 

II 

MARGINAL 

II 

UNACCEPTABLE 

II 

MARGINAL 

II 

UNACCEPTABLE 

II 

UNACCEPTABLE 

3 .  From  the  view  point  of  DIGITAL  CONTROL,  please 
of  one  rating  vs  the  others  in  the  table. 


cate  the  importcuice 
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4 .  From  the  viewpoint  of  RF  SIGNAL  PROCESSING  AND  SWITCHING,  please 


indicate  the  importance  of  one  rating  over  the  others  in  the  table. 
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II.  The  Following  Comparisons  are  related  to  SOUNDNESS  OF  APPROACH. 

1.  From  the  viewpoint  of  ANALYSIS  TASK,  please  indicate  the  in^ortance  of 


one  rating  over  the  others  in  the  table. 


2 .  From  the  view  point  of  DESIGN  TASK,  please  indicate  the  importance  of 
one  rating  vs  the  others  in  the  table . 
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III.  The  Following  Comparisons  are  related  to  COMPLIANCE  WITH  RQMTS. 

1.  From  the  viewpoint  of  ANALYSIS  TOPICS,  please  indicate  the  importance 
of  one  rating  over  the  others  in  the  table. 


2 .  From  the  view  point  of  DESIGN  CRITERIA,  please  indicate  the  importance 
of  one  rating  vs  the  others  in  the  table. 


3.  From  the  view  point  of  TEST  AND  DEMONSTRATION  REQUIREMENT,  please 
indicate  the  importance  of  one  rating  vs  the  others  in  the  table . 


EXCEPTIONAL 


ACCEPTABLE 


ACCEPTABLE 


MARGINAL 


UNACCEPTABLE 


MARGINAL 


UNACCEPTABLE 


UNACCEPTABLE 
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4 .  From  the  viewpoint  of  TECHNICAL  DISCUSSION,  please  indicate  the 
importance  of  one  rating  over  the  others  in  the  table . 


iimiiiiiiiiiiiii 

iiniiiiiiiiiiiiii 


ACCEPTABLE 


ACCEPTABLE 


MARGINAL 


ACCEPTABLE 


MARGINAL 


UNACCEPTABLE 


MARGINAL 


UNACCEPTABLE 


UNACCEPTABLE 


5 .  From  the  view  point  of  PRESENTATION,  please  indicate  the  importauice  of 
one  rating  vs  the  others  in  the  tcdjle. 
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IV.  The  Following  Comparisons  are  related  to  SPECIAL  TECH  FACTORS. 

1 .  From  the  viewpoint  of  SOLID  STATE  ARRAY  DESIGN,  please  indicate  the 
importance  of  one  rating  over  the  others  in  the  table. 


2.  From  the  view  point  of  ECM  SYSTEM  DEVELOPMENT, 
importcuice  of  one  rating  vs  the  others  in  the  table 


3.  From  the  view  point  of  CONTROLLER  DEVELOPMENT, 
importance  of  one  rating  vs  the  others  in  the  tedsle 


EXCEPTIONAL 


ACCEPTABLE 


ACCEPTABLE 


MARGINAL 


3 

I 

I 

I 

I 

I 

I 


ACCEPTABLE 


MARGINAL 


UNACCEPTABLE 


MARGINAL 


UNACCEPTABLE 


UNACCEPTABLE 


please  indicate  the 


I 

I 

I 

I 

I 

I 


ACCEPTABLE 


MARGINAL 


UNACCEPTABLE 


MARGINAL 


UNACCEPTABLE 


UNACCEPTABLE 


please  indicate  the 


ACCEPTABLE 


MARGINAL 


UNACCEPTABLE 


MARGINAL 


UNACCEPTABLE 


UNACCEPTABLE 
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4 .  From  the  viewpoint  of  ADVANCED  PROCESSING  INVESTIGATION,  please 


indicate  the  importance  of  one  rating  over  the  others  in  the  table. 


5.  From  the  view  point  of  EMBEDDED  SYSTEM  SOFTWARE  AND  ALGO  DEV,  please 
indicate  the  importance  of  one  rating  vs  the  others  in  the  table . 
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EVALUATION  OF  PROPOSALS 

The  ratings  in  the  tables  that  follow  are  to  be  used  to 
evaluate  the  proposals  against  the  criteria  previously 
developed.  The  ratings  provided  will  be  used  to  determine  the 
final  score  for  each  proposal.  For  a  given  table,  each  rating 
may  be  used  more  than  once  or  not  at  all. 
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I.  The  following  evaluations  are  related  to  UNDERSTANDING  THE  PROBLEM. 
1.  From  the  view  point  of  AIRCRAFT  REQUIREMENTS,  please  indicate  the 
rating  each  proposal  should  receive. 


EXCEPTIONAL 

ACCEPTABLE 

MARGINAL 

UNACCEPTABLE 

PROPOSAL  1 

PROPOSAL  2 

PROPOSAL  3 

2 .  From  the  view  point  of  ARRAY  CONFIGURATION,  please  indicate  the  rating 
each  proposal  should  receive . 


3.  From  the  view  point  of  DIGITAL  CONTROL,  please  indicate  the  rating 
each  proposal  should  receive. 


4 .  From  the  view  point  of  RF  SIGNAL  PROCESSING  AND  SWITCHING,  please 
indicate  the  rating  each  proposal  should  receive. 


PROPOSAL  3 
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5.  From  the  view  point  of  MISCELLANEOUS,  please  indicate  the  rating  each 
proposal  should  receive . 


II.  The  following  evaluations  are  related  to  SOUNDNESS  OF  APPROACH. 

1 .  From  the  view  point  of  ANALYSIS  TASK,  please  indicate  the  rating  each 
proposal  should  receive. 


2 .  From  the  view  point  of  DESIGN  TASK,  please  indicate  the  rating  each 
proposal  should  receive. 


3 .  From  the  view  point  of  TESTING  AND  DEMONSTRATION  TASK,  please  indicate 
the  rating  each  proposal  should  receive. 


PROPOSAL  3 


III.  The  following  evaluations  are  related  to  COMPLIANCE  WITH 

1.  From  the  view  point  of  ANALYSIS  TOPICS,  please  indicate  the  rating 

each  proposal  should  receive. 


EXCEPTIONAL  ACCEPTABLE  MARGINAL  UNACCEPTABLE 


PROPOSAL  1 


PROPOSAL  2 


PROPOSAL  3 


2.  From  the  view  point  of  DESIGN  CRITERIA,  please  indicate  the  rating 
each  proposal  should  receive. 


EXCEPTIONAL 

ACCEPTABLE 

MARGINAL 

UNACCEPTABLE 

PROPOSAL  1 


PROPOSAL  2 


PROPOSAL  3 


3.  From  the  view  point  of  TEST  AND  DEMONSTRATION  REQUIREMENT,  please 
indicate  the  rating  each  proposal  should  receive. 


EXCEPTIONAL 

ACCEPTABLE 

MARGINAL 

UNACCEPTABLE 

PROPOSAL  1 


PROPOSAL  2 


PROPOSAL  3 


4 .  From  the  view  point  of  TECHNICAL  DISCISSION,  please  indicate  the 
rating  each  proposal  should  receive. 


EXCEPTIONAL 

ACCEPTABLE 

MARGINAL 

UNACCEPTT^LE 

PROPOSAL  1 


PROPOSAL  2 


PROPOSAL  3 
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5 .  From  the  view  point  of  PRESENTATION,  please  indicate  the  rating  each 
proposal  should  receive . 


EXCEPTIONAL 

ACCEPTABLE 

MARGINAL 

UNACCEPTABLE 

PROPOSAL  1 

PROPOSAL  2 

PROPOSAL  3 

IV.  The  following  evaluations  are  related  to  SPBCIAI.  TECH  FACTORS. 

1 .  From  the  view  point  of  SOLID  STATE  ARRAY  DESIGN,  please  indicate  the 
rating  each  proposal  should  receive. 


EXCEPTIONAL 

ACCEPTABLE 

MARGINAL 

UNACCEPTABLE 

PROPOSAL  1 

PROPOSAL  2 

PROPOSAL  3 

2.  From  the  view  point  of  ECM  SYSTEM  DEVELOPMENT,  please  indicate  the 
rating  each  proposal  should  receive . 


EXCEPTIONAL 

ACCEPTABLE 

MARGINAL 

UNACCEPTABLE 

PROPOSAL  1 

PROPOSAL  2 

PROPOSAL  3 

3 .  From  the  view  point  of  DIGITAL  CONTROLLER  DEVELOPMENT,  please  indicate 
the  rating  each  proposal  should  receive. 


EXCEPTIONAL 

ACCEPTABLE 

PROPOSAL  1 

PROPOSAL  2 

PROPOSAL  3 
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4 .  From  the  view  point  of  ADVANCED  PROCESSING  INVESTIGATION,  please 
indicate  the  rating  each  proposal  should  receive . 


5 .  From  the  view  point  of  EMBEDDED  SYSTEM  SOFTWARE  AND  ALGORITHM 
DEVELOPMENT,  please  indicate  the  rating  each  proposal  should  receive . 


6.  From  the  view  point  of  MMIC  DEVICE  APPLICATION,  please  indicate  the 
rating  each  proposal  should  receive. 


